Chapter 1 · Risk and Frameworks
Risk Scoring, Controls and Decisions: Building a Toy Risk Register
- Page 2 of 22
- 21 min read
On the previous page you scoped the screening feature and wrote a register of seven risks. A register does not decide anything. Someone must still ask which controls really exist, whether a simpler option would do, how much risk is acceptable and who says so, and what must be true before a first small trial. This page turns the register into a decision a team can defend.
You need Python 3.12 and its standard library only. Each page runs in a fresh folder, so the first blocks below re-create, in compact form, the register from Impact Assessment: Scope, Stakeholders and Harms for a Bounded AI Feature. Type the blocks in order into one session; the text under a block is its real output.
Honesty convention. Every dataset is synthetic and labelled
Toy data:. Every company, person, rule and number is invented, including the thresholds. Results show how a calculation works; they say nothing about real people or real systems. Nothing here is legal advice.
What you will learn
- How to list existing controls honestly, with their limits, and separate planned controls from controls in place.
- How to compare non-AI options and human-only, AI-assisted and automated workflows before choosing.
- How to write risk-acceptance and escalation rules, set pilot conditions and name specialists, owners and reassessment triggers.
- How to scale an impact assessment to the risk, and how to reach a recorded decision for the screening feature.
Start from the register
This page covers the last steps of the decision pattern from page 1: options, decision with conditions, accountable person, review date. In NIST AI RMF 1.0 (January 2023, voluntary) terms it is closest to Manage (our gloss: decide what to do about the risks you mapped and measured).
First the record type: the page 1 record with controls moved to their own list, plus an owner and tags.
# file: risk.py
from dataclasses import dataclass
BANDS = ((16, "very high"), (10, "high"), (5, "medium"), (1, "low")) # our chosen cut-offs, as on page 1
def band(points):
return next(name for floor, name in BANDS if points >= floor)
@dataclass
class Risk:
id: str
title: str
level: str # individual or collective
likelihood: int
severity: int
reversibility: str # easy, hard, irreversible
exposed: int
evidence: str # none, opinion, internal-test, independent
res_likelihood: int = 0 # residual CLAIMED after controls; 0 = nothing claimed
res_severity: int = 0
owner: str = "" # a role, not "the team"
tags: tuple = () # what the risk touches: model, data, use, scale, vendor ...
def inherent(self):
return self.likelihood * self.severity
def claimed(self):
return self.res_likelihood * self.res_severity if self.res_likelihood else self.inherent()Now the data. The seven risks are those from page 1, with R7 completed as in that page's answer.
Toy data: seven risks and eight controls for the hypothetical screening feature.
from collections import namedtuple
from risk import Risk, band
# Toy data: the register from page 1, now with an owner and tags per risk. All invented.
RISKS = [Risk(**d) for d in [
dict(id="R1", title="Score penalises career breaks", level="individual", likelihood=3, severity=4,
reversibility="hard", exposed=900, evidence="opinion", res_likelihood=2, res_severity=3,
owner="Head of recruitment", tags=("model", "group")),
dict(id="R2", title="Recruiters trust the ranking", level="individual", likelihood=4, severity=3,
reversibility="hard", exposed=4800, evidence="internal-test", res_likelihood=3, res_severity=3,
owner="Head of recruitment", tags=("model", "use")),
dict(id="R3", title="Unusual CV layouts parse badly", level="individual", likelihood=3, severity=3,
reversibility="easy", exposed=600, evidence="internal-test", res_likelihood=3, res_severity=2,
owner="Operations manager", tags=("model", "data")),
dict(id="R4", title="CV text kept beyond hiring", level="individual", likelihood=2, severity=4,
reversibility="irreversible", exposed=6000, evidence="opinion",
owner="Operations manager", tags=("personal-data", "vendor", "data")),
dict(id="R5", title="Score reused to rank staff", level="individual", likelihood=2, severity=3,
reversibility="hard", exposed=250, evidence="none", res_likelihood=1, res_severity=3,
owner="Agency owner", tags=("use", "employment")),
dict(id="R6", title="Shortlists drift toward past hires", level="collective", likelihood=3, severity=3,
reversibility="hard", exposed=6000, evidence="opinion", res_likelihood=3, res_severity=2,
owner="the team", tags=("group", "scale", "model")),
dict(id="R7", title="Appeal route unclear", level="individual", likelihood=3, severity=3,
reversibility="hard", exposed=4800, evidence="none",
owner="Head of recruitment", tags=("use", "employment")),
]]
Control = namedtuple("Control", "id risk kind what status tested limit")
CONTROLS = [ # Toy data. status: in place or planned. tested: none, internal-test or independent
Control("C1", "R1", "prevent", "remove the career-gap feature", "in place", "none",
"other CV fields can stand in for a gap"),
Control("C2", "R1", "detect", "quarterly score check by group", "planned", "none",
"needs group labels the agency does not hold"),
Control("C3", "R2", "detect", "read 50 low-ranked CVs a month", "in place", "internal-test",
"50 of about 400 a month"),
Control("C4", "R3", "detect", "parse warning on odd layouts", "in place", "internal-test",
"sees only failures the parser notices"),
Control("C5", "R3", "correct", "manual check path", "in place", "none",
"the applicant must know to ask"),
Control("C6", "R5", "prevent", "written purpose limit", "in place", "none",
"a rule on paper, not a block"),
Control("C7", "R5", "detect", "access log", "in place", "internal-test",
"shows misuse after the fact"),
Control("C8", "R6", "detect", "yearly shortlist vs pool check", "planned", "none",
"slow: once a year"),
]
print(len(RISKS), "risks,", len(CONTROLS), "controls,", sum(c.status == "planned" for c in CONTROLS), "of them only planned")7 risks, 8 controls, 2 of them only plannedExisting controls and their limits
Identify existing controls and their limitations. A control is something that prevents a harm, detects it, or corrects it. Three questions separate a control from a hope: is it in place or only planned; has anyone tested it; and what is its limit, meaning what it cannot do. A detect control that nobody reads is not a control.
for c in CONTROLS:
print(f"{c.id} {c.risk} {c.kind:<8}{c.status:<10}tested: {c.tested:<14}{c.what}")
print(f" limit: {c.limit}")C1 R1 prevent in place tested: none remove the career-gap feature
limit: other CV fields can stand in for a gap
C2 R1 detect planned tested: none quarterly score check by group
limit: needs group labels the agency does not hold
C3 R2 detect in place tested: internal-test read 50 low-ranked CVs a month
limit: 50 of about 400 a month
C4 R3 detect in place tested: internal-test parse warning on odd layouts
limit: sees only failures the parser notices
C5 R3 correct in place tested: none manual check path
limit: the applicant must know to ask
C6 R5 prevent in place tested: none written purpose limit
limit: a rule on paper, not a block
C7 R5 detect in place tested: internal-test access log
limit: shows misuse after the fact
C8 R6 detect planned tested: none yearly shortlist vs pool check
limit: slow: once a yearPage 1 counted two controls for R1 and one for R6. But C2 and C8 are only planned, and C1 has never been tested. A planned control does not reduce risk today, so the next block recomputes the residual from controls in place only, and reports a doubt where a claim does not hold.
def in_place(risk):
return [c for c in CONTROLS if c.risk == risk.id and c.status == "in place"]
def supported(risk):
"""Residual points we can defend today: a planned control does not count."""
return risk.claimed() if in_place(risk) else risk.inherent()
def doubt(risk):
live = in_place(risk)
if risk.claimed() < risk.inherent() and not live:
return "claim rests on planned controls only"
if live and all(c.tested == "none" for c in live):
return "no in-place control has been tested"
return ""
print("id inherent claimed supported band doubt")
for r in RISKS:
s = supported(r)
print(f"{r.id:<4}{r.inherent():>6}{r.claimed():>9}{s:>10} {band(s):<11}{doubt(r)}")id inherent claimed supported band doubt
R1 12 6 6 medium no in-place control has been tested
R2 12 9 9 medium
R3 9 6 6 medium
R4 8 8 8 medium
R5 6 3 3 low
R6 9 6 9 medium claim rests on planned controls only
R7 9 9 9 mediumR6 goes back to 9, its inherent score, because its only control is planned. R1 keeps its claimed 6, with a doubt: its only live control, removing a field, may be defeated by a proxy (a different field that stands in for the removed one). R4 and R7 have no controls at all. These are findings, not accusations; the error would be to count a planned control.
Non-AI alternatives and three kinds of workflow
Evaluate non-AI alternatives before you compare AI designs. The honest question is "what problem are we solving, and is a simpler way enough?" Reasonable people disagree on the best option, because each moves cost and risk elsewhere.
| Option | What it gives | What it costs or risks |
|---|---|---|
| Keep reading by hand | A person reads every CV; no new risk from a score | Slow; tired readers are inconsistent; replies take longer |
| Short form with fixed questions | Less reading; answers are easy to compare | Some people find forms harder than CVs |
| Simple written rules (must hold a licence) | Everyone can see and challenge the rule | Rules can be crude; they need review too |
| AI ranking (the proposal) | Saves reading time at volume | The risks in the register; hard-to-see errors |
"Do not use AI" is a real option with its own harms, such as slower replies. Write the comparison down; do not try to win it.
Compare human-only, AI-assisted and automated workflows. The same AI score can sit in three different workflows. What changes is who makes the decision, and how many cases no person ever reads. The next block computes this with toy numbers: 6000 applications a year and 5 minutes to read one CV.
Toy data: three ways to run the same feature.
APPLICATIONS, MINUTES = 6000, 5 # Toy data: per year; minutes per CV
top = APPLICATIONS // 5
sample = (APPLICATIONS - top) // 20
WORKFLOWS = { # name: (CVs a person reads in full, who makes the call on a CV nobody read)
"human-only": (APPLICATIONS, "not applicable"),
"AI-assisted": (top + sample, "the ranking, by default"),
"automated": (0, "the cut-off, always"),
}
print(f"{'workflow':<13}{'read by a person':>17}{'hours':>7}{'never read':>12} who decides on those")
for name, (read, who) in WORKFLOWS.items():
print(f"{name:<13}{read:>17}{read * MINUTES / 60:>7.0f}{APPLICATIONS - read:>12} {who}")workflow read by a person hours never read who decides on those
human-only 6000 500 0 not applicable
AI-assisted 1440 120 4560 the ranking, by default
automated 0 0 6000 the cut-off, alwaysReading time falls from 500 hours to 120 hours in the assisted workflow, a real benefit. But 4560 applicants are never read. Whether that is acceptable depends on controls (such as a weekly sample), the cost of a missed good candidate, and whether applicants can ask for a review. "Assisted" is a promise about behaviour: if recruiters open only the top of the list, the workflow is automated under another name. Check what people do, not the label. The Beginner page Trust, Verification and Automation Bias: When to Rely on AI Output explains why.
Acceptance and escalation
Define risk-acceptance and escalation criteria. To accept a risk is to decide, knowingly and in writing, to live with it for a stated period. It does not make the risk go away. Escalation means passing the decision to someone with more authority when the risk is more than the owner should accept alone. Write the rule before the scores are seen, or it will bend to fit the answer someone wants. This invented rule is a choice, not a finding; your team might choose differently.
def route(r):
"""Our invented policy. Returns (route, reasons)."""
points = supported(r)
why = []
if band(points) == "very high":
return "stop", ["residual very high"]
if band(points) == "high":
why.append("residual high")
if r.reversibility == "irreversible":
why.append("irreversible")
if r.severity >= 4 and r.evidence in ("none", "opinion"):
why.append("severe+weak evidence")
if r.level == "collective":
why.append("collective")
if r.exposed >= 3000 and band(points) != "low":
why.append("wide reach")
return ("escalate" if why else "owner may accept"), why
for r in RISKS:
decision, why = route(r)
print(f"{r.id} {band(supported(r)):<8}{decision:<18}{'; '.join(why)}")R1 medium escalate severe+weak evidence
R2 medium escalate wide reach
R3 medium owner may accept
R4 medium escalate irreversible; severe+weak evidence; wide reach
R5 low owner may accept
R6 medium escalate collective; wide reach
R7 medium escalate wide reachFive of seven risks go up. Only R3 and R5 may be accepted by their owner. A reader may think five is too many or too few; that is a useful argument to have before the pilot. The rule "stop" never fired; it exists so that a very high residual score has an automatic consequence.
Conditions for a limited pilot
Set conditions for a limited pilot. A pilot is a small, time-boxed trial that limits who can be harmed while you learn. It needs a narrow scope, a review date, and for every condition a measure, a stop rule and an owner. Without them a condition is a wish. The checker below reports gaps and does not fill them.
Toy data: an invented pilot plan with two gaps left in on purpose.
PILOT = { # Invented plan
"scope": "one job family, 8 weeks, at most 600 applications, no automatic rejection",
"review_date": "2027-01-11",
"conditions": [ # (condition, measure, stop if, owner)
("read a random sample of low-ranked CVs every week", "wrongly low-ranked, per 50 read",
"3 or more", "Head of recruitment"),
("read the vendor contract: how long is CV text kept", "retention period written in R4",
"unknown at start", "Operations manager"),
("tell applicants that a tool orders the CVs", "notice text approved", "", "Agency owner"),
("answer every query about a rank within 5 working days", "", "", ""),
],
}
print("scope:", PILOT["scope"])
print("review:", PILOT["review_date"])
for n, (what, measure, stop, owner) in enumerate(PILOT["conditions"], 1):
gaps = [name for name, value in (("measure", measure), ("stop rule", stop), ("owner", owner)) if not value]
print(n, "complete" if not gaps else "MISSING " + ", ".join(gaps), "-", what)scope: one job family, 8 weeks, at most 600 applications, no automatic rejection
review: 2027-01-11
1 complete - read a random sample of low-ranked CVs every week
2 complete - read the vendor contract: how long is CV text kept
3 MISSING stop rule - tell applicants that a tool orders the CVs
4 MISSING measure, stop rule, owner - answer every query about a rank within 5 working daysThe stop rule "3 or more per 50 read" is a number the team chose; nothing here justifies it. Say so in the record and revisit it after the first weeks. Conditions 3 and 4 are unfinished, and the checker does not guess.
Specialists, owners and triggers
Identify risks that require specialist assessment. Some questions are not for a product team to answer, such as whether a contract term is acceptable, how long personal data may be kept, or whether a statistical comparison is sound. The team's job is to notice the question and ask early, not to answer it. For law, write "employment-law adviser: open", not your own conclusion.
SPECIALISTS = {
"group": "statistician or fairness reviewer",
"personal-data": "privacy or data-protection adviser",
"vendor": "contract reviewer",
"employment": "employment-law adviser",
}
for r in RISKS:
need = [SPECIALISTS[t] for t in r.tags if t in SPECIALISTS]
print(r.id, "; ".join(need) if need else "-")R1 statistician or fairness reviewer
R2 -
R3 -
R4 privacy or data-protection adviser; contract reviewer
R5 employment-law adviser
R6 statistician or fairness reviewer
R7 employment-law adviserMap risks to responsible owners. The owner of a risk is the person accountable for it: a role with a name behind it, not "the team". An owner can delegate the work but not the accountability.
from collections import Counter
VAGUE = ("", "the team", "everyone", "tbd")
load = Counter(r.owner for r in RISKS)
for owner, n in sorted(load.items()):
print(f"{owner:<22}owns {n}")
print("no named owner:", [r.id for r in RISKS if r.owner.lower() in VAGUE])Agency owner owns 1
Head of recruitment owns 3
Operations manager owns 2
the team owns 1
no named owner: ['R6']R6 has no named owner, so nobody is accountable for the one collective risk, and the Head of recruitment holds three risks and also runs the pilot. Both are things to fix or decide openly.
Define reassessment triggers after a change. A register is correct on the day it is written. It must be reopened when something changes that could make a score wrong: a new model version, a new data source, a new purpose or user group, a large change in volume, an incident or complaint, a failed control, or the review date. Reopen the risks the change touches; if the boundary moves (a new purpose), return to scoping.
Toy data: four invented changes and the risks their tags reopen.
TRIGGERS = [ # Invented examples: (what changed, which tags it touches)
("vendor ships a new scoring model", {"model", "vendor"}),
("applications double after a new office opens", {"scale"}),
("a manager asks to rank current staff", {"use"}),
("a new CV source brings a different file type", {"data"}),
]
for change, touched in TRIGGERS:
reopen = [r.id for r in RISKS if touched & set(r.tags)]
print(f"{change:<46}-> {', '.join(reopen)}")vendor ships a new scoring model -> R1, R2, R3, R4, R6
applications double after a new office opens -> R6
a manager asks to rank current staff -> R2, R5, R7
a new CV source brings a different file type -> R3, R4A new use reopens R5, but also R2 and R7, because it changes who is affected. A model update reopens five of seven risks. Triggers agreed in advance stop a team deciding "this change is too small" without looking.
A proportionate impact-assessment template
Apply a proportionate impact-assessment template. Proportionate means the effort fits the risk: a spell-checker needs a page, a screening tool needs more. One template, with parts you may skip when the risk is low, keeps this consistent.
Template:
IMPACT ASSESSMENT RECORD depth: light | standard | full version: __ date: ____-__-__
1 Situation feature, purpose in one sentence, boundary, out of scope
2 Who is affected users | affected non-users | other stakeholders
3 Benefits/harms benefits (how we would notice) | misuse | harms: individual and collective
4 Options do not use AI | non-AI alternative | human-only | AI-assisted | automated
5 Controls id | risk | kind | in place or planned | tested? | its limit
6 Risk register inherent | claimed residual | supported residual | evidence | assumptions
7 Decision approve | approve with conditions (pilot scope, measures, stop rules) | stop
8 Accountability escalation rule that fired | who decided, accepted until | specialists | owners
9 Review date | change triggers and the risks each one reopens| Depth | When | What you complete |
|---|---|---|
| Light | No decision about people; only low risks | Parts 1, 2, 7 and 9, on one page |
| Standard | Informs decisions about people, or some medium risks | All parts; owners may accept |
| Full | High residual risk, or irreversible or collective harm in decisions about people | All parts, plus specialist review and written sign-off above the owner |
The rule below is invented, like the other thresholds.
def depth(risks, about_people):
"""Our invented rule for how much assessment a feature needs."""
bands = {band(supported(r)) for r in risks}
hard_cases = any(r.reversibility == "irreversible" or r.level == "collective" for r in risks)
if bands & {"high", "very high"} or (about_people and hard_cases):
return "full"
if about_people or "medium" in bands:
return "standard"
return "light"
print("ShortlistHelper:", depth(RISKS, about_people=True))
typo = Risk("X1", "Typo in a reminder email", "individual", 2, 1, "easy", 100, "opinion")
print("reminder-email helper (invented, one low risk):", depth([typo], about_people=False))ShortlistHelper: full
reminder-email helper (invented, one low risk): lightExercise: assess the screening feature
The next block gathers the findings for ShortlistHelper in one place.
routes = {r.id: route(r) for r in RISKS}
gaps = [n for n, cond in enumerate(PILOT["conditions"], 1) if not all(cond)]
print("escalate:", [i for i, (d, _) in routes.items() if d == "escalate"])
print("owner may accept:", [i for i, (d, _) in routes.items() if d == "owner may accept"])
print("claims to re-check:", [r.id for r in RISKS if doubt(r)])
print("pilot conditions with gaps:", gaps)
print("risks without a named owner:", [r.id for r in RISKS if r.owner.lower() in VAGUE])
print("specialists to ask:", sorted({SPECIALISTS[t] for r in RISKS for t in r.tags if t in SPECIALISTS}))escalate: ['R1', 'R2', 'R4', 'R6', 'R7']
owner may accept: ['R3', 'R5']
claims to re-check: ['R1', 'R6']
pilot conditions with gaps: [3, 4]
risks without a named owner: ['R6']
specialists to ask: ['contract reviewer', 'employment-law adviser', 'privacy or data-protection adviser', 'statistician or fairness reviewer']This is evidence for a decision, not a verdict. Recruiters value saved time, applicants may value a human reading every CV, and the agency owner carries the legal and reputation risk. Deciding means choosing between these openly, with conditions. That is the Hard exercise below.
Where this shows up at work
Product, risk and procurement teams run this process whenever an AI feature goes to approval. "Escalate" may mean a committee in a large organisation and two people in a small one. The structure is the same: written rules, named roles, a date to look again.
Common mistakes
- Counting a planned control. It lowers the score on paper and nothing in practice. Count what is in place and tested; list the rest as planned.
- Accepting a risk with no date. It becomes permanent by accident. Write who accepted, until when, and what reopens it.
- Calling a workflow "assisted" while people follow the ranking. Measure what readers open; the label protects nobody.
- Owner is "the team". Nobody answers for a risk the team owns. Name a role.
- Pilot with no stop rule. A pilot that cannot stop is a rollout. Write the measure and the threshold before it starts.
Try it yourself
Use the namespace from this page. The Hard exercise revisits the decision pattern from page 1.
- Easy. A new risk, R8, "Rejection emails sent by mistake": likelihood 2, severity 5, hard to undo, 300 people exposed, evidence is opinion, no controls. Work out its points, band and route by hand, then check with code.
- Medium. Add a trigger for "an applicant says a rank was unfair" that touches the tags
modelandgroup. Which risks reopen? After you read the complaint, which register field would you update on the risks it supports? - Hard. Exercise: write the decision record for ShortlistHelper from the findings above: a decision (approve, approve with conditions, or stop), the conditions, what is not settled, who is accountable and when you review. Then say in two sentences what would change if the agency asked for the lowest-ranked 80% to be rejected automatically.
Answers
Easy. 2 x 5 is 10 points, which is "high" (10 or more). The route is "escalate" with two reasons: residual high, and severe with weak evidence (severity 5, opinion only).
r8 = Risk("R8", "Rejection emails sent by mistake", "individual", 2, 5, "hard", 300, "opinion")
print(r8.inherent(), band(r8.inherent()), route(r8))10 high ('escalate', ['residual high', 'severe+weak evidence'])Pass checklist: 10 points, the right band, both reasons found. A weak answer says "owner may accept" because only 300 people are exposed, ignoring severity.
Medium.
TRIGGERS.append(("an applicant says a rank was unfair", {"model", "group"}))
change, touched = TRIGGERS[-1]
print(change, "->", [r.id for r in RISKS if touched & set(r.tags)])an applicant says a rank was unfair -> ['R1', 'R2', 'R3', 'R6']R1, R2, R3 and R6 reopen. One complaint is one data point. If the team reproduces the problem, evidence can move from opinion to internal-test. Pass checklist: the four risks; a field named; one complaint is not treated as proof of a pattern. A weak answer changes a score because one person complained.
Hard. A strong answer is a conditional approval, not a rollout. One defensible record follows; another team could reasonably stop, or start with a non-AI option.
Decision record: ShortlistHelper pilot (invented) depth: full date: 2026-10-12 (invented)
Decision: approve with conditions: an 8-week pilot, one job family, at most 600 applications.
Not approved: rollout to all jobs; any automatic rejection; any use on current staff.
Why: saves reading time (about 500 h to about 120 h a year at full volume, toy numbers);
the unread 4560 are the main harm, so a weekly sample of low-ranked CVs is a condition.
Not settled: R4 retention (contract unread), R6 owner, privacy and employment-law review,
tests for the career-gap control (C1) and the group check (C2, only planned).
Accountable: Agency owner for the decision; Head of recruitment for the pilot conditions.
Review: 2027-01-11, or earlier if the vendor changes the model or the pilot stops itself.If the lowest-ranked 80% were rejected automatically, the workflow becomes "automated": the 4560 unread CVs become rejections with no person involved. The record should say "stop" or "escalate again" until a review route, a sample and a specialist opinion exist. Pass checklist: a clear decision with conditions; a measure and stop rule for the sample; what is not settled; a named accountable role; a review date; the automated variant treated as a new assessment. A weak answer approves unconditionally, or stops without saying what was weighed.
Summary
- List controls with status, test result and limit; count only controls in place, and report doubts instead of fixing claims.
- Compare non-AI options and human-only, AI-assisted and automated workflows, including how many cases no person reads.
- Write acceptance and escalation rules before seeing scores; thresholds are choices to justify.
- A pilot needs scope, a review date, and a measure, stop rule and owner for each condition.
- Name specialists, owners and reassessment triggers, and match the depth of assessment to the risk.
Next: Principles and Frameworks in Practice: From Values to Testable Controls turns shared principles into controls you can test, with a small mapping you can check in code.