Chapter 1 · Risk and Frameworks
Impact Assessment: Scope, Stakeholders and Harms for a Bounded AI Feature
- Page 1 of 22
- 21 min read
In the Beginner tutorial, Everyday AI Ethics: Who Benefits and Who Could Be Harmed, you asked those questions in your head or on a worksheet. In a team you need something repeatable: a written impact assessment that two people can read, challenge and update. This page teaches how to scope one for a small, bounded AI feature, and how to keep the risks in a register that a program can check.
You need Python 3.12 and its standard library only: no packages, no network, no model calls. Type the blocks in order into one session; they share variables, and the text under a block is its real output.
Honesty convention for this tutorial. Every dataset is synthetic and labelled
Toy data:. Every company, person and number is invented. A result shows how a calculation works; it says nothing about real people or real systems. Nothing here is legal advice.
What you will learn
- How to set a system boundary and purpose, and name users, affected non-users and other stakeholders.
- How to map an AI feature's inputs, outputs and decision points, and find the decision nobody makes.
- How to record inherent and residual risk, with likelihood, severity, reversibility, exposure and duration kept apart.
- How to tell individual from collective harm, build a risk matrix, and say honestly that its scores are a chosen method.
- How to record assumptions and evidence quality, and check a register in Python without inventing anything.
How to use this tutorial
The Advanced tutorial has six chapters, from risk and frameworks to fairness, documentation, safety, measurement and a final capstone. One running example carries through the first pages: a hypothetical applicant-screening feature.
The course decision pattern is below. Ethical concern, legal obligation and company policy are three different things; keep them apart when you write.
Template:
1 Situation what is proposed, for what purpose, inside which boundary
2 Who is affected users, affected non-users, other stakeholders
3 Benefits and harms intended benefits, plausible misuse, harms (individual and collective)
4 Options including "do not use AI" and "use it for less"
5 Evidence and unknowns what we measured, what we assume, how good the evidence is
6 Decision, conditions approve, approve with conditions, or stop
7 Who is accountable a named role, not "the team"
8 Review date when and what triggers a new lookFrameworks give this work a name. NIST AI RMF 1.0 (January 2023, voluntary) has four functions: Govern, Map, Measure and Manage. This page is closest to Map (our gloss: understand the context and the risks). Principles and Frameworks in Practice: From Values to Testable Controls returns to it.
Scope first: boundary and intended purpose
Define the system boundary and intended purpose. The system boundary says what you are assessing: the feature, the data it reads, the people who see its output, and the decisions it influences. The intended purpose is one sentence that says what the feature is for. Everything else is out of scope; write that down too, because features get used for jobs they were not built for.
Invented scenario:
Company: Harbourline Staffing, a small recruitment agency (invented).
Feature: ShortlistHelper reads the CVs sent for one job and gives each a "fit score" from 0 to 100.
Today: two recruiters read about 400 CVs per job by hand. The agency receives about 6000
applications a year (Toy data) and wants to save reading time.
Plan: the recruiter sees the CVs in score order and decides whom to interview.
Question: before anyone approves a pilot, what could go wrong, for whom, and how bad is it?A good purpose is narrow: "order the CVs for one job so a recruiter knows which to read first". A weak one is "improve hiring". The narrow one can be tested, and it shows what is outside. If someone later proposes rejecting people or ranking staff, that is a new feature and a new assessment.
Who is affected
Identify direct users, affected non-users and stakeholders. A user operates the feature. An affected non-user never touches it but is scored, ranked, watched or decided about. A stakeholder has an interest or responsibility without being either. The second group is easy to forget, and it often has the least power to object.
| Group | In the example | Why they matter |
|---|---|---|
| Direct users | Recruiters; the hiring manager who writes the job requirements | They act on the ranking and can override it |
| Affected non-users | Applicants; current employees whose CVs or scores could be reused | A low rank changes their chances; they never see why |
| Other stakeholders | Agency owner; the vendor; client employers; the local job market; a regulator, if one covers hiring | They carry cost, reputation or legal duty |
Benefits and plausible misuse
Document intended benefits and plausible misuse. Write the benefits first and honestly: here, recruiters spend less time reading and replies may arrive sooner. A benefit counts only if you can say how you would notice it. Then write plausible misuse: use that is not intended but easy and likely, such as a manager asking for the score to rank current staff, or an applicant filling a CV with job-ad keywords. Misuse is not an accusation; it is a prediction about a tool that is easy to reuse.
Reasonable people weigh this differently. A recruiter with 400 CVs for one job may value saved time most; an applicant may value a human reading every CV. An assessment does not settle that; it records both and makes the trade-off visible to whoever decides.
Map inputs, outputs and decision points
Map the AI feature's inputs, outputs and decision points. Draw the flow as data. List what goes in, what comes out, and every point where something is decided, with who decides. A small structure is enough, and it can be checked by a program.
Toy data: the feature map for the screening example.
FEATURE = {
"name": "ShortlistHelper (hypothetical)",
"purpose": "order the CVs for one job so a recruiter knows which to read first",
"out_of_scope": ["rejecting anyone", "ranking current staff", "pay or promotion"],
"flow": [ # (step, what happens, who decides)
("1 input", "applicant uploads a CV", "applicant"),
("2 input", "hiring manager lists job requirements", "hiring manager"),
("3 AI", "feature gives each CV a fit score 0-100", "model"),
("4 output", "CVs are shown as a ranked list", "feature"),
("5 decision", "recruiter opens the top 20% of CVs", "recruiter"),
("6 decision", "the other 80% are never opened", "nobody (default)"),
("7 decision", "recruiter invites people to interview", "recruiter"),
],
}
print(FEATURE["name"], "-", FEATURE["purpose"])
print("out of scope:", ", ".join(FEATURE["out_of_scope"]))
for step, what, who in FEATURE["flow"]:
print(f"{step:<11}{what:<44}{who}")
silent = [s for s, _, who in FEATURE["flow"] if who.startswith("nobody")]
print("steps where no person decides:", silent)ShortlistHelper (hypothetical) - order the CVs for one job so a recruiter knows which to read first
out of scope: rejecting anyone, ranking current staff, pay or promotion
1 input applicant uploads a CV applicant
2 input hiring manager lists job requirements hiring manager
3 AI feature gives each CV a fit score 0-100 model
4 output CVs are shown as a ranked list feature
5 decision recruiter opens the top 20% of CVs recruiter
6 decision the other 80% are never opened nobody (default)
7 decision recruiter invites people to interview recruiter
steps where no person decides: ['6 decision']Steps 5 and 7 are decisions by a person. Step 6 is a decision too, even though no one makes it: 80% of applicants are never read because of where the model placed them. This is where an AI score becomes a human outcome. The Beginner idea of automation bias (trusting a ranking because it looks authoritative) lives here.
One risk, one record
A risk register is a list of risk records. The fields below carry the concepts of this page; the sections after the code explain each one.
| Field | Records |
|---|---|
likelihood, severity | Inherent risk: before any control, each on a 1-5 scale defined by your team |
reversibility | Whether the harm can be undone: easy, hard, irreversible. Recorded on its own, never hidden inside a product |
exposed, duration | How many people per year, and for how long (one-off, weeks, persistent) |
level | Individual harm or collective harm |
controls, res_likelihood, res_severity | Existing measures and the residual risk after them |
assumptions, evidence | What you took for granted and how good the support is |
Create this file. It uses only the standard library, and the validator reports what is missing; it never fills a gap with a guess.
# file: risk.py
from dataclasses import dataclass, fields
LEVELS = ("individual", "collective")
REVERSIBILITY = ("easy", "hard", "irreversible")
DURATION = ("one-off", "weeks", "persistent")
EVIDENCE = ("none", "opinion", "internal-test", "independent")
BANDS = ((16, "very high"), (10, "high"), (5, "medium"), (1, "low")) # our chosen cut-offs
OPTIONAL = ("controls", "res_likelihood", "res_severity")
@dataclass
class Risk:
id: str
title: str = ""
level: str = "" # individual or collective harm
who: str = ""
likelihood: int = 0 # 1-5, inherent: before any control
severity: int = 0 # 1-5, inherent
reversibility: str = "" # recorded separately, not multiplied in
exposed: int = 0 # people per year
duration: str = ""
controls: tuple = ()
res_likelihood: int = 0 # residual: after the controls above; 0 = not recorded
res_severity: int = 0
assumptions: tuple = ()
evidence: str = ""
def residual(self):
if not self.controls:
return self.likelihood, self.severity
return self.res_likelihood, self.res_severity
def score(likelihood, severity):
return likelihood * severity
def band(points):
return next(name for floor, name in BANDS if points >= floor)
def problems(r):
"""Missing or invalid fields. Nothing is guessed or filled in."""
found = [f"{f.name} is missing" for f in fields(r)
if f.name not in OPTIONAL and not getattr(r, f.name)]
for name, allowed in (("level", LEVELS), ("reversibility", REVERSIBILITY),
("duration", DURATION), ("evidence", EVIDENCE)):
value = getattr(r, name)
if value and value not in allowed:
found.append(f"{name} '{value}' is not one of {allowed}")
for name in ("likelihood", "severity", "res_likelihood", "res_severity"):
value = getattr(r, name)
if value and not 1 <= value <= 5:
found.append(f"{name} {value} is outside 1-5")
if r.controls and not (r.res_likelihood and r.res_severity):
found.append("controls listed but residual scores missing")
if not r.controls and (r.res_likelihood or r.res_severity):
found.append("residual scores given but no controls listed")
if all(r.residual()) and score(*r.residual()) > score(r.likelihood, r.severity):
found.append("residual is higher than inherent: check")
return found
def caveats(r):
"""Valid records can still be weak."""
notes = []
if r.evidence == "none":
notes.append("scores rest on assumption only")
if r.evidence == "opinion" and r.severity >= 4:
notes.append("severe risk with opinion-only evidence")
return notes
def table(risks):
rows = sorted(risks, key=lambda r: (-score(r.likelihood, r.severity), -r.severity, r.id))
out = [f"{'id':<4}{'level':<12}{'L':>2}{'S':>3} {'rev':<13}{'exposed':>8} "
f"{'duration':<11}{'pts':>3} {'band':<10}evidence"]
for r in rows:
pts = score(r.likelihood, r.severity)
out.append(f"{r.id:<4}{r.level:<12}{r.likelihood:>2}{r.severity:>3} {r.reversibility:<13}"
f"{r.exposed:>8} {r.duration:<11}{pts:>3} {band(pts):<10}{r.evidence}")
return "\n".join(out)
def matrix(risks, residual=False):
cell = {}
for r in risks:
key = r.residual() if residual else (r.likelihood, r.severity)
cell.setdefault(key, []).append(r.id)
lines = ["sev\\lik" + "".join(f"{n:^9}" for n in range(1, 6))]
for s in range(5, 0, -1):
lines.append(f"{s:^7}" + "".join(f"{','.join(cell.get((n, s), ['.'])):^9}" for n in range(1, 6)))
return "\n".join(lines)The band cut-offs (16, 10, 5) are ones we chose. They sit in one constant so a team can change them and record why.
Inherent and residual risk
Distinguish inherent risk from residual risk. Inherent risk is the risk with no controls. Residual risk is what remains after the controls you actually have, not the ones you plan. Recording both shows what the controls buy. In the code, a risk without controls has residual equal to inherent; a risk with controls must state the new scores, and a residual score higher than the inherent one is flagged as a probable mistake.
Likelihood, severity and reversibility
Estimate likelihood, severity and reversibility as three separate answers. Each scale needs anchors written down, or two people will use the same number for different things.
| Score | Likelihood (in a year of use) | Severity (for one person) |
|---|---|---|
| 1 | Would surprise the team | Minor annoyance, fixed at once |
| 3 | Could happen at some point in the year | Lost opportunity or money, with effort to repair |
| 5 | Expected most months | Serious, lasting damage to livelihood, health or rights |
Reversibility asks a different question: if it happens, can it be put right? A wrongly delayed reply is easy. A rejection nobody notices is hard, because the person cannot ask for what they do not know happened. Leaked CV data is irreversible.
Exposure, scale and duration
Identify exposure, scale and duration of potential harm. Exposure is how many people could be touched, scale is how large the effect is for each, and duration is how long it lasts. A small harm to everyone can outweigh a large harm to a few. The register stores the number exposed per year; the table below shows it beside the duration. R2 touches 4800 of the 6000 applicants (80%) because of the cut-off in step 6; R5 touches 250 (about 4%). A persistent harm to 4800 people is a very different thing from a harm that lasts weeks for 600, even if one person's experience looks similar.
Individual and collective harm
Distinguish individual harm from collective or societal harm. Individual harm falls on identifiable people: a CV scored down, data kept too long. Collective harm appears across many people and may be invisible in any one case: if a shortlist drifts toward the profile of past hires, no single applicant can show it, yet over years the pool of people who get interviews narrows. Record the level of each risk so collective risks are not lost. They need different evidence: one complaint shows an individual harm; only a comparison across many cases shows a collective one.
Fill the register and check it
Now put invented risks for the screening feature into records: six complete ones and one incomplete record, R7, with an impossible score and a made-up evidence label. The checker must report R7's gaps and nothing else.
Toy data: seven invented risks and their checks.
from risk import Risk, problems, caveats
RAW = [ # Toy data: invented risks for the hypothetical screening feature, 6000 applicants a year
dict(id="R1", title="Score penalises career breaks", level="individual",
who="applicants with a career break", likelihood=3, severity=4, reversibility="hard",
exposed=900, duration="persistent", controls=("remove gap feature", "quarterly group check"),
res_likelihood=2, res_severity=3, evidence="opinion",
assumptions=("15% of applicants have a break (guess, not measured)",)),
dict(id="R2", title="Recruiters trust the ranking", level="individual",
who="applicants ranked below the cut", likelihood=4, severity=3, reversibility="hard",
exposed=4800, duration="persistent", controls=("sample 50 low-ranked CVs a month",),
res_likelihood=3, res_severity=3, evidence="internal-test",
assumptions=("recruiters open only the top 20% of the list",)),
dict(id="R3", title="Unusual CV layouts parse badly", level="individual",
who="applicants with non-standard CVs", likelihood=3, severity=3, reversibility="easy",
exposed=600, duration="weeks", controls=("show a parse warning", "manual check path"),
res_likelihood=3, res_severity=2, evidence="internal-test",
assumptions=("10% of CVs do not follow the template (guess)",)),
dict(id="R4", title="CV text kept beyond hiring", level="individual",
who="all applicants", likelihood=2, severity=4, reversibility="irreversible",
exposed=6000, duration="persistent", evidence="opinion",
assumptions=("vendor keeps text for 90 days (contract not yet read)",)),
dict(id="R5", title="Score reused to rank staff", level="individual",
who="current employees", likelihood=2, severity=3, reversibility="hard",
exposed=250, duration="persistent", controls=("written purpose limit", "access log"),
res_likelihood=1, res_severity=3, evidence="none",
assumptions=("nobody has asked for this yet",)),
dict(id="R6", title="Shortlists drift toward past hires", level="collective",
who="future applicants, local job market", likelihood=3, severity=3, reversibility="hard",
exposed=6000, duration="persistent", controls=("yearly shortlist vs pool comparison",),
res_likelihood=3, res_severity=2, evidence="opinion",
assumptions=("past hires are not a balanced sample (guess)",)),
dict(id="R7", title="Appeal route unclear", likelihood=3, severity=7, evidence="rumour"),
]
register = [Risk(**d) for d in RAW]
for r in register:
issues = problems(r)
print(r.id, "INCOMPLETE" if issues else "complete")
for line in issues:
print(" problem:", line)
for line in caveats(r):
print(" caveat: ", line)
valid = [r for r in register if not problems(r)]R1 complete
caveat: severe risk with opinion-only evidence
R2 complete
R3 complete
R4 complete
caveat: severe risk with opinion-only evidence
R5 complete
caveat: scores rest on assumption only
R6 complete
R7 INCOMPLETE
problem: level is missing
problem: who is missing
problem: reversibility is missing
problem: exposed is missing
problem: duration is missing
problem: assumptions is missing
problem: evidence 'rumour' is not one of ('none', 'opinion', 'internal-test', 'independent')
problem: severity 7 is outside 1-5R7 is not scored and not repaired: it is refused, with the reasons. This is the rule for any documentation checker: report what is missing, never invent evidence. A program that fills the blanks produces a register that looks finished and means nothing. The caveats are a second kind of message: R1 and R4 are severe risks supported only by opinion, and R5 rests on assumption alone.
A risk matrix, and what it is not
Build a risk matrix without treating scores as objective truth. A matrix puts likelihood on one axis and severity on the other and places each risk in a cell.
from risk import table, matrix
print(f"{len(valid)} of {len(register)} records complete; R7 stays out until it is fixed\n")
print(table(valid))
print()
print(matrix(valid))6 of 7 records complete; R7 stays out until it is fixed
id level L S rev exposed duration pts band evidence
R1 individual 3 4 hard 900 persistent 12 high opinion
R2 individual 4 3 hard 4800 persistent 12 high internal-test
R3 individual 3 3 easy 600 weeks 9 medium internal-test
R6 collective 3 3 hard 6000 persistent 9 medium opinion
R4 individual 2 4 irreversible 6000 persistent 8 medium opinion
R5 individual 2 3 hard 250 persistent 6 medium none
sev\lik 1 2 3 4 5
5 . . . . .
4 . R4 R1 . .
3 . R5 R3,R6 R2 .
2 . . . . .
1 . . . . .R1 and R2 both score 12, in different cells: the table breaks the tie by severity, a rule we chose. Now see what the controls did.
from risk import score
print("id inherent residual controls")
for r in valid:
print(f"{r.id:<4}{score(r.likelihood, r.severity):>7}{score(*r.residual()):>10} {len(r.controls)}")
print()
print("residual matrix")
print(matrix(valid, residual=True))id inherent residual controls
R1 12 6 2
R2 12 9 1
R3 9 6 2
R4 8 8 0
R5 6 3 2
R6 9 6 1
residual matrix
sev\lik 1 2 3 4 5
5 . . . . .
4 . R4 . . .
3 R5 R1 R2 . .
2 . . R3,R6 . .
1 . . . . .R4 had no control and does not move; it is now the only severity 4 risk left. That is a finding a table of totals hides.
Warning. These scores are a chosen assessment method, not measurements. The scales, the multiplication and the bands are decisions made by people, and another reasonable method gives another order. The numbers help discussion and consistency; they are not truth, and never replace judgement about a single severe case.
To see this, rank the same six risks three ways: product of likelihood and severity; severity first; and product with severity raised by one when the harm is hard to undo.
from risk import score
STEP = {"easy": 0, "hard": 1, "irreversible": 1} # a chosen rule: raise severity by one if hard to undo
METHODS = {
"likelihood x severity": lambda r: score(r.likelihood, r.severity),
"severity first": lambda r: (r.severity, r.likelihood),
"L x S, severity +1 if hard to undo": lambda r: score(r.likelihood, min(5, r.severity + STEP[r.reversibility])),
}
for name, key in METHODS.items():
order = sorted(sorted(valid, key=lambda r: r.id), key=key, reverse=True) # ties keep id order
print(f"{name:<36}", " ".join(r.id for r in order))likelihood x severity R1 R2 R3 R6 R4 R5
severity first R1 R4 R2 R3 R6 R5
L x S, severity +1 if hard to undo R2 R1 R6 R4 R3 R5R4 (leaked or retained CV data) is fifth under the first rule and second under the second. R6 (collective drift) rises when reversibility counts. None of these orders is wrong. A team should pick one before it sees the results, write the reason down, and look at the risks that move.
Assumptions and evidence quality
Document assumptions and evidence quality. Every score depends on assumptions such as "15% of applicants have a career break (guess, not measured)". Write them next to the score, with the label "guess" where it is one. The evidence field says how good the support is: none, opinion, internal-test (the team measured something on its own data) or independent (a review by someone outside the team). Weak evidence does not mean a low score: it means the score is uncertain, and a severe risk with weak evidence is a reason to learn more before deciding.
Where this shows up at work
Product and risk teams keep a register for each AI feature, review it before a pilot and update it after a change. A useful habit: when someone says "the risk is low", ask "which record, which assumption, which evidence?".
Common mistakes
- A boundary that says "the model". The harm happens in the workflow around it, such as the unread 80%. Assess the feature with its people and decisions.
- Forgetting affected non-users. If only users are listed, the people with most to lose are missing. Ask who is decided about without being asked.
- Treating a score as a measurement. Two people can score honestly and differ. Record the scale, the method and who agreed it.
- Multiplying away reversibility. A hard-to-undo harm can look small as a product. Keep it as its own field and look at it.
- Filling gaps to make the register look complete. An honest "missing" or "evidence: none" is more useful than an invented figure.
Try it yourself
Use the namespace from this page. The exercises revisit Where Bias Comes From: Data, Design, Proxies and Feedback Loops from the Beginner tutorial.
- Easy. Without running code, place a new risk with likelihood 4 and severity 5 on the matrix. What are its points and band? Then name one way a "fit score" could treat a group unfairly (for example through a proxy variable) and which register field would record it.
- Medium. R7 was refused. Complete it for the case where the agency has no written appeal route, without inventing evidence, and run the checker on it.
- Hard. Use the
METHODSdictionary from this page. Write code that prints every risk whose rank changes by two places or more between methods, with the ranks.
Answers
Easy. 4 x 5 is 20 points, so "very high" (16 or more), in the top row, fourth column of the matrix. A proxy variable (such as a career break standing in for age or caregiving) is a good example: it would be recorded in who (the group affected), assumptions (the claim that the feature acts as a proxy) and level (individual or collective).
Pass checklist: 20 points and the right band; one concrete mechanism named and tied to a field; a note that the answer depends on the team's own scale. A weak answer gives a verdict on fairness from the score alone.
Medium.
r7 = Risk(id="R7", title="Appeal route unclear", level="individual", who="rejected applicants",
likelihood=3, severity=3, reversibility="hard", exposed=4800, duration="persistent",
assumptions=("no appeal route is written down yet",), evidence="none")
print(problems(r7), caveats(r7))[] ['scores rest on assumption only']Pass checklist: every required field is filled with something the scenario supports; evidence is "none" because nothing was measured; the checker returns no problems and one caveat. A weak answer sets evidence to "internal-test" to avoid the caveat.
Hard.
ids = sorted(r.id for r in valid)
rank = {r_id: [] for r_id in ids}
for key in METHODS.values():
order = sorted(sorted(valid, key=lambda r: r.id), key=key, reverse=True)
for position, r in enumerate(order, 1):
rank[r.id].append(position)
for r_id in ids:
spread = max(rank[r_id]) - min(rank[r_id])
if spread >= 2:
print(r_id, "ranks", rank[r_id], "spread", spread)R2 ranks [2, 3, 1] spread 2
R3 ranks [3, 4, 5] spread 2
R4 ranks [5, 2, 4] spread 3
R6 ranks [4, 5, 3] spread 2Pass checklist: ranks match the previous block; R4 has the widest spread. A strong answer adds that these are the risks to discuss, not to ignore.
Summary
- Scope the feature narrowly, write what is out of scope, and list users, affected non-users and stakeholders.
- Map inputs, outputs and decision points; look for a decision that no person makes.
- Record inherent and residual risk, and keep likelihood, severity, reversibility, exposure, duration and level as separate fields.
- A risk matrix is a chosen method: rank by more than one rule and discuss the risks that move.
- Write assumptions and evidence quality next to every score; a checker reports gaps and never invents evidence.
Next: Risk Scoring, Controls and Decisions: Building a Toy Risk Register adds controls, alternatives, acceptance rules and reassessment triggers to this register.