Chapter 5 · Quality, Release and the Future
Software Quality, Testing and QA
- Page 15 of 18
- 20 min read
ClinicQueue is almost ready. Patients can book a slot, the SMS reminder works on Farhan's laptop, and Rina has seen the queue screen in a demo. Nadia asks the obvious question: "Is it good enough to switch on for real patients?" Nobody can answer that with a feeling. The answer comes from quality work done all through the project and from testing that is planned, not improvised.
On the previous page, Code Review, Version Control and Team Workflow, you saw how a team keeps the main branch releasable. This page is about how the team knows it really is releasable. You will separate quality assurance from testing, meet the four test levels and the test pyramid, design test cases with two classic techniques and run them with Python's built-in unittest, write a useful bug report, and see how to test a feature whose output is written by an AI model. Every ClinicQueue number and rule here is invented.
What you will learn
- The difference between quality assurance, quality control and testing, and between verification and validation.
- The four common test levels (unit, integration, system, acceptance) and the idea of the test pyramid.
- How to choose test cases with equivalence partitioning and boundary values, and run them with
assertandunittest. - How to write a good bug report, why severity and priority are different, and how a defect moves through its life cycle.
- How to test an AI feature when you cannot list every possible output.
Quality, QA and testing
"Quality" sounds vague, but on earlier pages you already made it concrete: the functional requirements say what ClinicQueue must do, the non-functional requirements (from the ISO/IEC 25010 checklist on the requirements pages) say how well, and the acceptance criteria say how Nadia will check it. Quality means the product meets those needs. Three activities protect it, and people often mix their names up.
QA, QC and testing
The ISTQB Certified Tester Foundation Level syllabus (v4.0.1), an international syllabus for software testers, draws the line like this. Quality assurance (QA) "is a process-oriented, preventive approach": it improves the way the team works so that fewer defects are made in the first place, and it "is the responsibility of everyone on a project". Testing "is a product-oriented, corrective approach": it examines the product to find defects so they can be fixed. Testing is "a major form of quality control (QC)", the family of activities that check the product itself.
| Quality assurance | Quality control (including testing) | |
|---|---|---|
| Looks at | The process: how the team works | The product: what the team built |
| Aim | Prevent defects | Find defects so they can be fixed |
| ClinicQueue example | Every pull request needs a review; every story has acceptance criteria before work starts; the team agrees a Definition of Done | Kwame runs system tests on the booking flow; Farhan's unit tests run on every push; a reviewer reads the diff |
| Who | Everyone | Testers, developers, reviewers, and the customer at acceptance |
So Kwame is not "the quality person" who catches everyone's mistakes at the end; he is a testing specialist on a team where everybody owns quality.
Verification and validation
ISTQB defines verification as "checking whether the system meets specified requirements" and validation as "checking whether the system meets users' and other stakeholders' needs in its operational environment". The usual memory aid: verification asks "did we build it right?", validation asks "did we build the right thing?"
Invented scenario:
Requirement REQ-05: "The queue screen shall list today's patients in order of appointment time."
Verification: Kwame checks the screen with 12 test bookings. They appear in time order.
REQ-05 passes.
Validation: Rina uses the screen for one real morning at the front desk. Patients who arrive
early are served early, so the time order is not the order she actually calls them.
She needs an "arrived" mark and a second list. REQ-05 was met, but the need was not.Both checks are needed: software can match what was written and still fail its users. That is why acceptance testing with real users, in their real setting, belongs in every plan.
Error, defect, failure
Testers use three words carefully. In the ISTQB wording: "Human beings make errors (mistakes), which produce defects (faults, bugs), which in turn may result in failures." Farhan misreads the rule (an error), writes < instead of <= (a defect, or bug), and a patient cannot book 14 days ahead (a failure, the wrong behaviour someone sees). Some defects never cause a failure (the code never runs), and some failures have other causes, such as a full disk.
Seven principles worth knowing
The same syllabus lists seven testing principles. You do not need to memorise them, but each one changes how you plan tests:
| Principle | What it means for ClinicQueue |
|---|---|
| 1 Testing shows the presence, not the absence of defects | "All tests passed" means no test found a bug, not that there are none. |
| 2 Exhaustive testing is impossible | You cannot try every date, time and phone number, so you choose test cases with techniques (next sections). |
| 3 Early testing saves time and money | Reviewing the acceptance criteria before coding finds problems when they cost a conversation, not a rewrite. |
| 4 Defects cluster together | If the date handling had three bugs, look harder there for a fourth. |
| 5 Tests wear out | Running the same tests forever finds fewer new bugs; add new cases when the code changes. |
| 6 Testing is context dependent | A clinic booking app needs different testing from a game or a payroll system. |
| 7 Absence-of-defects fallacy | A bug-free queue screen that Rina cannot use is still a failure (the validation story above). |
Test levels and the test pyramid
Tests are grouped into levels by how much of the system they exercise at once. The roadmap's four levels are the common names; ISTQB calls unit testing "component testing" and also splits integration into component integration and system integration testing.
| Level | What it checks | ClinicQueue example | Usually done by |
|---|---|---|---|
| Unit (component) | One function or class on its own | can_book(14) returns True | Developers, automated |
| Integration | Parts working together | The booking API saves a row in the database and the SMS module picks it up | Developers and testers |
| System | The whole product, end to end, against the requirements | Kwame books, cancels and rebooks in a browser on a test server and checks the SMS and the queue screen | Testers |
| Acceptance | Whether the customer and users accept it for real use | Nadia and Rina run the Given/When/Then criteria from the requirements pages on the test server | Customer and users, with the team |
The acceptance criteria you wrote earlier in Given/When/Then form are already acceptance tests in waiting. "Given a slot is free, when a patient books it, then the slot shows as taken for everyone else" becomes a concrete check at the acceptance level and, often, a smaller automated test lower down.
The test pyramid
Mike Cohn's test pyramid (he first drew it around 2003 to 2004 and called it the "Test Automation Pyramid" in his 2009 book Succeeding with Agile) gives one rule of shape: have many more fast, low-level tests than slow, broad ones.
Example:
/\
/ \ few end-to-end / UI tests
/ UI \ (slow, break easily, but test what users see)
/------\
/ \ some integration / service tests
/ service \ (API + database + SMS stub)
/------------\
/ \ many unit tests
/ unit \ (milliseconds each, run on every push)
/------------------\Why this shape? A unit test of can_book runs in a fraction of a second and, when it fails, points at one function. A browser test of the whole booking flow takes much longer, can fail because a page loaded slowly, and only tells you "something in booking broke". The pyramid is about balance, not a fixed ratio; no reliable rule gives a percentage per layer. Some practitioners give integration tests a bigger share in apps that mostly connect services. Either way, do not rely mostly on slow end-to-end tests.
Choosing test cases: partitions and boundaries
Principle 2 says you cannot test everything. Black-box techniques choose test cases from the specification alone, without looking at the code. Two of the most useful work together.
ClinicQueue has an invented rule from the requirements: a patient can book a slot from 1 to 14 days ahead. Today (0 days ahead) is not allowed, because the front desk plans each day the evening before.
Equivalence partitioning
Split the possible inputs into groups (partitions) that the software should treat the same way. If one value from a group works, the others very likely work too, so one test per partition is enough to start.
| Partition | Values | Expected | One test value |
|---|---|---|---|
| Too early | 0 and below | Refused | -3 |
| Allowed | 1 to 14 | Accepted | 7 |
| Too late | 15 and above | Refused | 30 |
Boundary value analysis
Defects love edges: < written instead of <=, a count that starts at 0 instead of 1. So add test values on each side of every edge between partitions. Here the edges are between 0 and 1 and between 14 and 15, which gives four boundary values: 0, 1, 14 and 15. The middle value 7 would never catch a slip at the edge; the boundary values will.
A first check can use plain assert, which stops with an AssertionError if a condition is false:
MAX_DAYS_AHEAD = 14
def can_book(days_ahead):
"""ClinicQueue rule: a slot can be booked from 1 to 14 days ahead."""
return 1 <= days_ahead <= MAX_DAYS_AHEAD
# quick checks with plain assert: one value per partition, then the boundaries
assert can_book(-3) is False # partition "too early" (the past)
assert can_book(7) is True # partition "allowed"
assert can_book(30) is False # partition "too late"
for days in [0, 1, 14, 15]:
print(days, can_book(days))0 False
1 True
14 True
15 FalseThe three asserts pass silently (one value per partition), and the loop shows the four boundary values behave as the rule says.
Turning the checks into a unittest suite
Loose asserts are fine for a quick check, but a team wants tests that live next to the code, run with one command and report each case by name. Python's standard library includes unittest for that. You write a class that inherits from unittest.TestCase; every method whose name starts with test is one test case, and methods like assertTrue and assertFalse report a clear message when they fail. Each message says what the rule expects, so a failure explains itself.
The small run_tests helper runs the class and prints one line per test plus a short summary. (Normally you would run the tests from the command line; the helper keeps the output short and identical on every run, without timings.)
import io
import unittest
class TestCanBook(unittest.TestCase):
def test_today_is_too_early(self):
self.assertFalse(can_book(0), "0 days ahead (today) must be refused")
def test_first_allowed_day(self):
self.assertTrue(can_book(1), "1 day ahead is the first allowed day")
def test_middle_of_range(self):
self.assertTrue(can_book(7), "7 days ahead must be allowed")
def test_last_allowed_day(self):
self.assertTrue(can_book(14), "14 days ahead is the last allowed day")
def test_first_day_too_late(self):
self.assertFalse(can_book(15), "15 days ahead must be refused")
def run_tests(case):
"""Run one TestCase class; print one line per test and a summary without timings."""
suite = unittest.defaultTestLoader.loadTestsFromTestCase(case)
names = [test.id() for test in suite] # read before running: the suite empties itself
result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=2).run(suite)
problems = {test.id(): trace for test, trace in result.failures + result.errors}
for name in names:
short = name.split(".")[-1]
if name in problems:
print(f"FAIL {short}: {problems[name].strip().splitlines()[-1]}")
else:
print(f"ok {short}")
print("all passed:", result.wasSuccessful(), "| tests run:", result.testsRun)
run_tests(TestCanBook)ok test_first_allowed_day
ok test_first_day_too_late
ok test_last_allowed_day
ok test_middle_of_range
ok test_today_is_too_early
all passed: True | tests run: 5unittest runs the methods in alphabetical order of their names, so good tests never depend on each other's order.
The tests earn their keep: a regression
Two weeks later, Aiko tidies the booking module and rewrites the rule in a way that looks equivalent. The existing tests run again on her change, as they do on every push. Running old tests after a change, to make sure nothing that used to work is now broken, is called regression testing.
def can_book(days_ahead):
"""Refactored version with a slip: < instead of <=."""
return 0 < days_ahead < MAX_DAYS_AHEAD
run_tests(TestCanBook)ok test_first_allowed_day
ok test_first_day_too_late
FAIL test_last_allowed_day: AssertionError: False is not true : 14 days ahead is the last allowed day
ok test_middle_of_range
ok test_today_is_too_early
all passed: False | tests run: 5Only the boundary test at 14 fails; the middle-of-range test still passes. Without the boundary value, the bug would have reached patients as "you cannot book two weeks ahead". The failure message names the rule, so Aiko knows what to fix before anyone has to explain it. She changes the comparison back to <=, the five tests pass again, and the code reviewer sees the tests in the pull request.
Tests are code too: name them after the rule they protect, keep them small, and review them. A test that never fails, whatever you break, tests nothing.
Bugs: reporting and tracking
When a test or a user finds a failure, the next step is a bug report (ISTQB calls it a defect report) in the team's issue tracker. A good report lets a developer reproduce the problem without asking a single question.
Template:
Title: Booking refused for exactly 14 days ahead
ID / date: CQ-212, 2026-11-20, reported by Kwame
Environment: test server, build 0.9.3, desktop browser
Steps: 1. Open the booking page as a patient
2. Choose Dr. Rahman and a free slot on 2026-12-04 (14 days after 2026-11-20)
3. Press "Book"
Expected: Booking is saved (rule: 1 to 14 days ahead is allowed)
Actual: Message "This date cannot be booked"; no booking saved
Evidence: screenshot attached; 13 days ahead works, 15 days ahead is refused correctly
Severity: medium (a valid booking is blocked; patients can still book other days)
Priority: high (fix before go-live)
Status: openWhat makes it good: one problem per report, exact steps, expected versus actual, the environment and version, and neighbouring values that narrow down the cause. Avoid "booking is broken", blame ("Aiko's change broke it") and several problems in one report.
Severity versus priority
ISTQB lists these as separate fields: severity is the "degree of impact" of the defect, and priority is how urgently to fix it. They often go together, but not always, and the difference is a common interview question.
| Case | Severity | Priority | Why |
|---|---|---|---|
| Reminders go to the wrong patient's phone | High | High | Harm to patients and private data exposed; stop and fix now |
| The yearly statistics export crashes; it is next needed in eleven months | High | Low | The feature is fully broken, but nobody needs it soon |
| The clinic's name is misspelt on the booking page | Low | High | Nothing breaks, but every patient sees it on launch day |
| A tooltip on an admin screen has a typo | Low | Low | Fix when someone is working on that screen |
The tester usually proposes the severity, because it is a technical judgement. Priority is a business decision, so the Product Owner (on ClinicQueue, Nadia) has the final word.
The defect life cycle
A defect report moves through statuses until it is closed. Issue trackers use different names; the statuses below are the examples ISTQB gives.
Example:
open --+--> waiting to be fixed --(fix)--> awaiting confirmation testing --(passes)--> closed
| ^ |
| | (fails)
| +------------ re-opened <-------------+
+--> rejected (works as intended, not a bug)
+--> duplicate (already reported)
+--> deferred (fix in a later release)Reading it: a new report is open. The team may reject it (it is the intended behaviour), mark it a duplicate, defer it to a later release, or accept it as waiting to be fixed. After the fix, the tester runs a confirmation test (the original steps again). If it passes, the report is closed; if not, it is re-opened and goes back to the developer. Add a regression test for the bug at the same time, so it cannot come back unnoticed.
Testing an AI feature
ClinicQueue's later feature asks an AI model to draft the reminder text for each booking. That breaks the habit you just learned: for can_book(14) there is exactly one right answer, but a model can write the same reminder in thousands of acceptable ways, and it may answer differently each time. You cannot write "expected output equals this sentence".
What you can do is test rules that every acceptable output must follow, and run them on a sample of real-looking inputs:
- It must contain the patient's first name, the date and the time of this booking.
- It must stay under the team's length limit (200 characters, an invented ClinicQueue rule).
- It must not mention medical details, such as test results or medicines, because SMS messages can be read by others.
# Booking details and three drafts produced by the AI feature (all invented)
booking = {"first_name": "Mitu", "date": "2026-12-03", "time": "10:30"}
drafts = [
"Hello Mitu, this is a reminder of your ClinicQueue appointment on 2026-12-03 at 10:30.",
"Hi Mitu! See you tomorrow at 10:30. Please bring your blood test results.",
"Dear patient, your appointment is on 2026-12-03 at 10:30. Reply STOP to opt out.",
]
BANNED_WORDS = ["test results", "diagnosis", "medicine"] # no medical details in an SMS
def check_reminder(text, booking, max_length=200):
"""Return a list of rule violations for one drafted reminder (empty list = passes)."""
problems = []
for field in ["first_name", "date", "time"]:
if booking[field] not in text:
problems.append(f"missing {field}")
if len(text) > max_length:
problems.append(f"too long ({len(text)} characters)")
for word in BANNED_WORDS:
if word in text.lower():
problems.append(f"medical detail: '{word}'")
return problems
for number, text in enumerate(drafts, 1):
problems = check_reminder(text, booking)
print(f"draft {number}:", "ok" if not problems else "; ".join(problems))draft 1: ok
draft 2: missing date; medical detail: 'test results'
draft 3: missing first_nameDraft 1 passes. Draft 2 says "tomorrow" instead of the date (a problem if the SMS is delayed) and adds a medical detail the model invented. Draft 3 forgot the name. In the real feature, a failed draft is not sent: ClinicQueue falls back to a fixed template text, and the failure is logged so the team can see how often it happens. Drafts that pass still go to a person at the clinic for approval, as planned from the start; because ClinicQueue fills in the date and time itself (Risk Management), these checks are a second line of defence.
Rule checks catch what code can detect. They cannot judge tone or politeness, so Kwame also reads a sample of drafts by hand every week, and the team keeps a fixed set of test bookings (a patient with a long name, an appointment just after midnight, a name in Bangla script) to re-run whenever the prompt or the model changes. This builds on how Prompt Engineering for AI: Beginner teaches you to check AI output: write down what a good answer must contain, keep a saved test input with its expected output, and re-test when the prompt or the model changes. Here the same checks become code, so they can run on many examples, not one.
Where this shows up at work
- Most teams run the unit and integration tests automatically on every push or pull request; a red test blocks the merge.
- Testers write test cases from the requirements and acceptance criteria before the code is finished, which often exposes unclear requirements early.
- Bug triage meetings, often weekly, set priority on new reports with the product owner.
- Teams with AI features keep a set of evaluation examples and re-run it whenever the prompt or the model changes.
Common mistakes
- Testing only the happy path. One booking 7 days ahead proves almost nothing. Fix: one test per partition, plus the boundary values on each side of every edge.
- Leaving testing to the end and to one person. Defects pile up and the release slips. Fix: unit tests with every change, testers involved from the requirements, and QA as everyone's job.
- Vague bug reports. "Booking is broken" costs a round of questions. Fix: steps, expected, actual, environment and version, one problem per report, no blame.
- Mixing up severity and priority. Then urgent cosmetic problems wait and harmless crashes jump the queue. Fix: record both separately and let the business decide priority.
- Testing an AI feature with one example that "looked fine". Fix: write rules every output must meet, check them on a sample, and keep a fallback for drafts that fail.
Try it yourself
- Easy: ClinicQueue lets a patient add a note to a booking of 0 to 300 characters (0 means no note). List the equivalence partitions and the boundary values you would test, with the expected result for each.
- Medium: Classify each defect by severity and priority (high or low), with one sentence of reasoning: (a) the queue screen freezes every morning at 09:00 when the clinic opens; (b) the Bangla translation of the "Cancel" button is missing on the settings page used only by Nadia; (c) the SMS reminder is sent twice to some patients.
- Hard: Nadia adds a rule: "a patient can have at most 2 future bookings at a time". Write a function
can_add_booking(future_bookings)and aunittest.TestCasewith boundary tests, then run it withrun_testsfrom this page. Also name one test at a level above unit that this rule needs.
Answers
- Partitions: below 0 (impossible for a length, so it is not a real input and needs no test), 0 to 300 accepted, 301 and above refused. Boundary values: 0 (accepted, no note), 1 (accepted), 300 (accepted), 301 (refused). One value from the middle, such as 150 (accepted), represents the valid partition. A good answer notices that the lower edge is a real special case (an empty note must not cause an error) and tests both sides of the upper edge.
- (a) High severity, high priority: the main screen of the front desk stops at the busiest moment of every day. (b) Low severity, low priority: one rarely used screen for one person who reads English; fix it with the next settings change (a team that promised full Bangla support might raise the priority). (c) High severity, high priority: patients are annoyed and may stop trusting reminders, and every extra SMS costs money. Different answers can be right if the reasoning separates impact (severity) from urgency (priority).
- Model answer:
def can_add_booking(future_bookings): return future_bookings < 2. Tests:test_none_yetasserts True for 0,test_one_bookingasserts True for 1 (the last value that allows one more),test_two_bookingsasserts False for 2 (the first value that refuses). Runningrun_testson the class prints three "ok" lines and "all passed: True | tests run: 3". A good answer tests both sides of the edge between 1 and 2, gives each assertion a message that states the rule, and adds a system or integration test, for example "book twice, cancel one, then a third booking succeeds", because the count must come from the database and must update after a cancellation.
Summary
- QA is process-oriented and preventive and belongs to everyone; testing is product-oriented, a major form of quality control. Verification checks the specification; validation checks the real need.
- Test at four levels (unit or component, integration, system, acceptance) and, following Mike Cohn's test pyramid, keep many more fast low-level tests than slow broad ones, without a fixed ratio.
- Choose test cases with equivalence partitions and boundary values; automate them with
unittestso they run as regression tests on every change. - A good bug report is reproducible and blame-free; severity (impact) and priority (urgency) are separate; a fix is closed only after a confirmation test.
- For AI features, test rules that every output must meet, on samples, with a fallback and regular human review.
Next: Releasing and Maintaining Software: how ClinicQueue gets a version number, reaches real patients safely, and stays healthy for years after launch.

