Measurement and Risk-Control Tests: Do the Controls Actually Work?

A control that nobody tests is a hope. The earlier pages produced risk registers, documentation and governance records; this page asks whether the controls inside them actually work, and how you would know. It turns vague aims into objectives with a number and an owner, plans a small test, builds a toy dataset, computes group measures, produces a register and a draft system card from structured records, and ends with a small test harness whose answers are PASS, FAIL or NOT TESTED.

You need Python 3.12 and its standard library only: no packages, no network, no model calls. Type the blocks in order into one session; they share variables, and the text under a block is its real output.

Honesty convention. Every dataset is synthetic and labelled Toy data:. Every organisation, person, threshold and number is invented. A result shows how a calculation or a test works; it says nothing about real people or real systems, and a PASS on toy data proves nothing about a real service. Scripts report MISSING or NOT TESTED where information is absent and never invent it. Nothing here is legal advice.

What you will learn

• How to turn "be fair" into a measurable objective with a measure, a threshold, an owner and an action if missed, and how leading indicators differ from outcome measures.
• How to write a small risk-control test plan, and how to build an offline toy dataset that can be checked by hand.
• How to calculate group measures with the standard library, and to flag small samples and missing groups instead of reporting them.
• How to generate a risk register and a draft system card from structured records, and validate an impact assessment, without inventing fields.
• How to test whether a control meets its acceptance criteria and report PASS, FAIL or NOT TESTED.

The running example

Invented scenario:

Organisation: Orchid Training Fund (invented), a non-profit with 60 subsidised course places.
Feature:      PlaceSorter reads applications and marks each "invite" or "hold"; a staff member
              reads every "invite" first and decides.
Groups:       applicants are grouped by region of application: north, south, east, west
              (placeholders; the fund's policy names all four).
Label:        "reviewers judged the applicant eligible" in a later full read (1 or 0).
Question:     do the controls the fund wrote down work, and can a program show it?

The previous page, Review, Audit and Governance in Practice: Change Approval, Risk Acceptance and Staff Concerns, gave each system an owner and a change-approval rule. Here those owners need numbers to look at, and the approvers need to know what a failed test means.

Measurable objectives

Define measurable responsible-use objectives. "Be fair" cannot be checked. A measurable objective names what is measured, on whom, over what period, against which limit, who owns it and what happens when the limit is missed. The limit is a choice by people; nothing here is a standard.

Two cautions. A number can steer people toward the number: if staff are judged on a low complaint count, complaints may get harder to make. And much that matters cannot be counted well, so keep a few written, qualitative reviews beside the measures.

Leading indicators and outcome measures

Distinguish leading indicators from outcome measures. A leading indicator tells you early whether the conditions for good results are in place. An outcome measure tells you what actually happened to people. You need both: outcomes are the point, but they arrive late and are often rare; leading indicators are quick, but can drift away from outcomes.

A low override rate shows the trap: it may mean the tool is good, or that reviewers have stopped questioning it (see Human Oversight Design: Meaningful Review, Not Rubber-Stamping). Check from time to time that each leading indicator still moves with the outcome it stands for.

A small risk-control test plan

Design a small risk-control test plan. A plan says, before any test is run, what the control claims, how it will be checked, on what data, and what counts as a pass. Writing the criterion first prevents choosing it after seeing the result. These controls come from the fund's risk register.

The thresholds and the reference group are decisions. Say who may change them: raising a limit to make a test pass is a change that needs the approval described on the previous page.

An offline toy dataset

Create an offline toy dataset for a risk demonstration. Build the data from counts so that every number can be checked by hand, then shuffle it with a fixed seed so the order looks like real data and still repeats exactly.

Toy data: 116 invented applications; west has none.

import random

OBJ = {"min_each": 10, "max_selection_gap": 0.10, "max_tpr_gap": 0.10}   # chosen limits, not standards
CELLS = [  # Toy data: (group, label 1=eligible, pred 1=invite, how many people)
    ("north", 1, 1, 24), ("north", 1, 0, 6), ("north", 0, 1, 8), ("north", 0, 0, 32),
    ("south", 1, 1, 10), ("south", 1, 0, 8), ("south", 0, 1, 4), ("south", 0, 0, 18),
    ("east", 1, 1, 2), ("east", 1, 0, 1), ("east", 0, 1, 0), ("east", 0, 0, 3),
]
EXPECTED = ["north", "south", "east", "west"]   # the groups the fund's policy names

def build(seed=7):
    rows = [dict(group=g, label=l, pred=p) for g, l, p, n in CELLS for _ in range(n)]
    random.Random(seed).shuffle(rows)
    return rows

ROWS = build()
print(len(ROWS), "rows;", ROWS[0], "...")
print({g: sum(r["group"] == g for r in ROWS) for g in EXPECTED})

116 rows; {'group': 'north', 'label': 0, 'pred': 0} ...
{'north': 70, 'south': 40, 'east': 6, 'west': 0}

Group measures in a few lines

Calculate fairness metrics with the standard library. The meaning of these measures, and the arguments about which to prefer, are in Fairness Metrics on Toy Data: Selection Rates and Error Rates in Python and Comparing Fairness Goals and Their Limits: Small Samples and Conflicting Metrics. Here only the calculation is re-created, compactly, so that a test can call it: selection rate (share invited), true-positive rate (share of eligible applicants invited) and false-positive rate (share of ineligible applicants invited). A rate with an empty denominator is None, never zero.

def rates(rows):
    out = {}
    for g in sorted({r["group"] for r in rows}):
        s = [r for r in rows if r["group"] == g]
        tp = sum(r["label"] == 1 and r["pred"] == 1 for r in s)
        fn = sum(r["label"] == 1 and r["pred"] == 0 for r in s)
        fp = sum(r["label"] == 0 and r["pred"] == 1 for r in s)
        neg = len(s) - tp - fn
        out[g] = dict(n=len(s), pos=tp + fn, neg=neg, selection=(tp + fp) / len(s),
                      tpr=tp / (tp + fn) if tp + fn else None, fpr=fp / neg if neg else None)
    return out

def f2(x):
    return "n/a" if x is None else f"{x:.2f}"

R = rates(ROWS)
print(f"{'group':<7}{'n':>4}{'elig':>6}{'inelig':>8}{'selection':>11}{'tpr':>6}{'fpr':>6}")
for g, v in R.items():
    print(f"{g:<7}{v['n']:>4}{v['pos']:>6}{v['neg']:>8}{v['selection']:>11.2f}{f2(v['tpr']):>6}{f2(v['fpr']):>6}")

group     n  elig  inelig  selection   tpr   fpr
east      6     3       3       0.33  0.67  0.00
north    70    30      40       0.46  0.80  0.20
south    40    18      22       0.35  0.56  0.18

Small samples and missing groups

Flag small sample sizes and missing groups. East has three eligible and three ineligible people: one different decision would move its true-positive rate by a third. West has no rows at all, which could mean nobody applied or that the data never arrived; the program cannot tell, so it must not treat silence as fairness. The rule below runs before any gap is shown. Its minimum is the chosen min_each limit; the second of those two pages shows a way to put an interval around a small rate.

def status(R, expected=EXPECTED, minimum=OBJ["min_each"]):
    out = {}
    for g in expected:
        if g not in R:
            out[g] = "MISSING: no rows for this group"
        elif min(R[g]["pos"], R[g]["neg"]) < minimum:
            out[g] = f"NOT REPORTED: {R[g]['pos']} eligible, {R[g]['neg']} ineligible (minimum {minimum})"
        else:
            out[g] = "ok"
    return out

def gaps(R, st, ref="north"):
    return {g: {k: R[g][k] - R[ref][k] for k in ("selection", "tpr", "fpr")}
            for g in R if st.get(g) == "ok" and g != ref}

ST = status(R)
G = gaps(R, ST)
for g, s in ST.items():
    print(f"{g:<7}{s}")
for g, d in G.items():
    print(f"gap {g} - north:", ", ".join(f"{k} {v:+.3f}" for k, v in d.items()))

north  ok
south  ok
east   NOT REPORTED: 3 eligible, 3 ineligible (minimum 10)
west   MISSING: no rows for this group
gap south - north: selection -0.107, tpr -0.244, fpr -0.018

The reference group (north, the largest) is a convenience for the arithmetic, not a ranking of groups. The gaps for south are visible; east and west produce no number at all, and that silence is reported rather than hidden.

A risk register from structured records

Generate a risk register from structured records. Findings from tests and reviews arrive as records. A program can turn them into a register, sort them and flag what is not ready, but a person must rate likelihood and severity: the program does not guess a rating. Scores and bands follow the method of Impact Assessment: Scope, Stakeholders and Harms for a Bounded AI Feature and Risk Scoring, Controls and Decisions: Building a Toy Risk Register; the cut-offs are again our choice.

FINDINGS = [  # Toy data: likelihood and severity (1-5) are rated by a person; None = not yet rated
    dict(id="F1", text="Eligible south applicants invited less often (tpr gap)", likelihood=4,
         severity=4, rated_by="Selection lead", owner="Head of Selection"),
    dict(id="F2", text="East group too small to judge", likelihood=3, severity=3,
         rated_by="Selection lead", owner=""),
    dict(id="F3", text="No test rows for west", likelihood=None, severity=None, rated_by="",
         owner="Data lead"),
]
BANDS = ((16, "very high"), (10, "high"), (5, "medium"), (1, "low"))

def register(findings):
    rated, waiting = [], []
    for f in findings:
        if f["likelihood"] and f["severity"] and f["rated_by"]:
            pts = f["likelihood"] * f["severity"]
            rated.append((pts, f["severity"], f["id"], next(b for lo, b in BANDS if pts >= lo), f))
        else:
            waiting.append(f)
    return sorted(rated, key=lambda x: (-x[0], -x[1], x[2])), waiting

rated, waiting = register(FINDINGS)
for pts, _, fid, band, f in rated:
    note = "" if f["owner"] else "  [MISSING owner]"
    print(f"{fid}  {pts:>2} pts  {band:<9}  {f['text']}{note}")
for f in waiting:
    print(f"{f['id']}  NOT RATED - a person must rate likelihood and severity: {f['text']}")

F1  16 pts  very high  Eligible south applicants invited less often (tpr gap)
F2   9 pts  medium     East group too small to judge  [MISSING owner]
F3  NOT RATED - a person must rate likelihood and severity: No test rows for west

A draft system card, and a checked impact assessment

Generate a draft system card from a Python dictionary, and validate required fields in an impact assessment. One dictionary holds the assessment. A validator lists required fields that are empty; the card draft prints what the dictionary says, marks the rest MISSING and attaches the measured results, or the reason a result is not reported. The longer, more careful version of this idea is in Disclosures and Documentation Validation: Checking a System Card with Python.

IA = dict(  # Toy data: one invented impact assessment, partly filled in
    purpose="order applications so staff read the likeliest eligible first",
    out_of_scope=["automatic rejection"], affected=["applicants", "selection staff"],
    harms=["eligible applicants invited less often in some regions"],
    controls=[], residual_risk="", owner="Head of Selection", review_date="2027-03-01")
REQUIRED = ("purpose", "out_of_scope", "affected", "harms", "controls", "residual_risk",
            "owner", "review_date")

def missing(d, required=REQUIRED):
    return [k for k in required if not d.get(k)]

def card(d, R, st):
    lines = ["SYSTEM CARD DRAFT (not reviewed)"]
    for k in REQUIRED:
        v = d.get(k)
        lines.append(f"{k}: " + ("MISSING" if not v else "; ".join(v) if isinstance(v, list) else v))
    lines.append("measured on toy data:")
    for g, s in st.items():
        ok = f"selection {R[g]['selection']:.2f}, tpr {f2(R[g]['tpr'])}" if s == "ok" else s
        lines.append(f"  {g}: {ok}")
    return "\n".join(lines)

gone = missing(IA)
print(f"{len(gone)} of {len(REQUIRED)} required fields missing: {', '.join(gone)}\n")
print(card(IA, R, ST))

2 of 8 required fields missing: controls, residual_risk

SYSTEM CARD DRAFT (not reviewed)
purpose: order applications so staff read the likeliest eligible first
out_of_scope: automatic rejection
affected: applicants; selection staff
harms: eligible applicants invited less often in some regions
controls: MISSING
residual_risk: MISSING
owner: Head of Selection
review_date: 2027-03-01
measured on toy data:
  north: selection 0.46, tpr 0.80
  south: selection 0.35, tpr 0.56
  east: NOT REPORTED: 3 eligible, 3 ineligible (minimum 10)
  west: MISSING: no rows for this group

The draft is honest about its holes: no controls and no residual risk are written, so the card says MISSING and does not supply plausible sentences. Completing those fields needs the people who know the answers.

Does the control meet its criterion?

Test whether a control meets its acceptance criteria. Each test below is a function written with assert: it passes silently or raises AssertionError with a message. A harness runs them and reports one of three results. PASS means the criterion was met on this data. FAIL means it was not. NOT TESTED means no test exists or the data needed was absent; it is never counted as a pass. (Python skips assert statements when run with the -O option, so in real projects use a test framework or raise errors explicitly.)

class NotTested(Exception):
    pass

def c1():
    assert "east" not in G, "a gap was reported for a group below the minimum"

def c2():
    worst = max(abs(d["selection"]) for d in G.values())
    assert worst <= OBJ["max_selection_gap"], f"selection gap {worst:.3f} is above {OBJ['max_selection_gap']:.2f}"

def c3():
    worst = max(abs(d["tpr"]) for d in G.values())
    assert worst <= OBJ["max_tpr_gap"], f"true-positive-rate gap {worst:.3f} is above {OBJ['max_tpr_gap']:.2f}"

def c4():
    absent = [g for g in EXPECTED if g not in R]
    assert not absent, "no rows for: " + ", ".join(absent)

def c6():
    gone = missing(IA)
    assert not gone, "missing fields: " + ", ".join(gone)

def c7():
    assert rates(build(7)) == rates(build(7)), "same seed gave different tables"

CONTROLS = [("C1", "Small groups are not compared", c1), ("C2", "Selection-rate gap at most 0.10", c2),
            ("C3", "True-positive-rate gap at most 0.10", c3), ("C4", "Every policy group is tested", c4),
            ("C5", "A named person approves each release", None), ("C6", "Impact assessment complete", c6),
            ("C7", "Test data rebuilds exactly", c7)]

def run(controls):
    tally = {"PASS": 0, "FAIL": 0, "NOT TESTED": 0}
    for cid, text, test in controls:
        try:
            if test is None:
                raise NotTested("no program test written")
            test()
            result, why = "PASS", ""
        except AssertionError as e:
            result, why = "FAIL", str(e)
        except NotTested as e:
            result, why = "NOT TESTED", str(e)
        tally[result] += 1
        print(f"{cid} {result:<11}{text}" + (f"\n      why: {why}" if why else ""))
    print(tally)

run(CONTROLS)

C1 PASS       Small groups are not compared
C2 FAIL       Selection-rate gap at most 0.10
      why: selection gap 0.107 is above 0.10
C3 FAIL       True-positive-rate gap at most 0.10
      why: true-positive-rate gap 0.244 is above 0.10
C4 FAIL       Every policy group is tested
      why: no rows for: west
C5 NOT TESTED A named person approves each release
      why: no program test written
C6 FAIL       Impact assessment complete
      why: missing fields: controls, residual_risk
C7 PASS       Test data rebuilds exactly
{'PASS': 2, 'FAIL': 4, 'NOT TESTED': 1}

Read each line as a prompt for a decision, not a verdict on the tool. C2 fails by a small margin (0.107 against 0.10) on a few dozen invented rows: that is a reason to look closer and to ask how sure the number is, not proof of unfairness. C3 fails by a larger margin. C4 fails because west has no data, and C5 is NOT TESTED because approval is a record someone must read. C1 and C7 pass, which only says those two pieces of machinery work on this data. A failed control triggers the change-approval and review steps described on the previous page; the outcome is recorded either way.

Where this shows up at work

Analysts and product teams run a small set of such checks before each release and every quarter, and keep the output next to the version it tested. Reviewers and auditors then read a list of PASS, FAIL and NOT TESTED rather than a general statement of confidence. The habit that matters most is writing the pass criterion before looking at the data.

Common mistakes

1. Setting the threshold after seeing the result. A limit chosen to fit the number proves nothing. Write it first, with the owner who may change it.
2. Counting NOT TESTED as a pass. An untested control is an unknown. Report it as such and plan the test.
3. Reporting a rate for a tiny or empty group. A third of three people is not a finding, and silence for a missing group is not fairness. Flag both.
4. Trusting a leading indicator alone. A tidy review log may sit beside real harm. Compare it with an outcome measure from time to time.
5. Reading toy results as facts about the world. The data here are invented to show a calculation. Real data needs real governance, domain review and the questions from the earlier pages.

Try it yourself

Use the namespace from this page. The tasks revisit the governance register from Review, Audit and Governance in Practice: Change Approval, Risk Acceptance and Staff Concerns.

1. Easy. Raise max_selection_gap to 0.12 and re-run only C2. What changes, and why is this not a good way to fix a failed test?
2. Medium. Add a control C8: "false-positive-rate gap at most 0.10" and run it with the others.
3. Hard. Add C9: "true-positive-rate gap for east is at most 0.10". The control must report NOT TESTED when east has too few eligible applicants, and say why.

Answers

Easy.

OBJ["max_selection_gap"] = 0.12
run([c for c in CONTROLS if c[0] == "C2"])
OBJ["max_selection_gap"] = 0.10   # put the chosen limit back

C2 PASS       Selection-rate gap at most 0.10
{'PASS': 1, 'FAIL': 0, 'NOT TESTED': 0}

Pass checklist: C2 now passes; the answer says the limit was changed after the result, that a change like this needs a recorded reason and the approval for the system's tier, that the data did not change, and that the label still says 0.10 although the limit is now 0.12. A weak answer calls the system fair because the test passes.

Medium.

def c8():
    worst = max(abs(d["fpr"]) for d in G.values())
    assert worst <= 0.10, f"false-positive-rate gap {worst:.3f} is above 0.10"

run([("C8", "False-positive-rate gap at most 0.10", c8)])

C8 PASS       False-positive-rate gap at most 0.10
{'PASS': 1, 'FAIL': 0, 'NOT TESTED': 0}

Pass checklist: a test function with a clear message; the result is PASS, with the remark that it covers one reported group only. A weak answer hard-codes the answer instead of computing it.

Hard.

def c9():
    if R["east"]["pos"] < OBJ["min_each"]:
        raise NotTested(f"east has {R['east']['pos']} eligible applicants, minimum {OBJ['min_each']}")
    assert abs(R["east"]["tpr"] - R["north"]["tpr"]) <= OBJ["max_tpr_gap"], "east tpr gap too large"

run([("C9", "East true-positive-rate gap at most 0.10", c9)])

C9 NOT TESTED East true-positive-rate gap at most 0.10
      why: east has 3 eligible applicants, minimum 10
{'PASS': 0, 'FAIL': 0, 'NOT TESTED': 1}

Pass checklist: NOT TESTED, not PASS and not FAIL, because the rate would rest on three people; the reason names the numbers. A strong answer adds the next step: collect more data, or accept that this group cannot be tested yet and record that decision and who made it.

Summary

• A measurable objective names the measure, the threshold, the owner and the action if missed; use leading indicators and outcome measures together.
• Write the test plan and its pass criteria before running anything, and keep the thresholds under change approval.
• Flag small samples and missing groups instead of reporting rates for them; silence is not evidence of fairness.
• Generate registers and cards from records, but let people rate and fill; report MISSING, never invent.
• A harness gives PASS, FAIL or NOT TESTED; NOT TESTED is an unknown, and toy results illustrate a calculation only.

Next: Incidents and Continuous Improvement: Triage, Post-Mortems and Review Cycles covers what to do when a control fails in real use.