Fairness Metrics on Toy Data: Selection Rates and Error Rates in Python

The risk register on the first pages of this tutorial holds claims such as "the score penalises career breaks". A claim like that can be tested. This page starts the second chapter, Fairness and Data: you build a small table of invented decisions and compute, group by group, who is selected and who is wrongly selected or missed. You also learn what those numbers cannot tell you.

You need Python 3.12, standard library only. Type the blocks in order into one session; they share variables, and the text under a block is its real output.

Honesty convention. The dataset is synthetic and labelled Toy data:; every person, group and number is invented. Results show how a calculation works. They are not evidence about any real system or any real group of people.

What you will learn

• How to define the decision, the outcome and the comparison groups before computing anything, and why protected attributes and proxy variables need separate checks.
• How to keep labels, predictions and observed outcomes apart, and what goes missing when only outcomes are observed.
• How to compute selection rates, false-positive and false-negative rates, accuracy, precision and recall per group.
• How to compute demographic parity, equal opportunity and equalised odds differences, and why a gap is not a verdict.
• What fairness metrics on a table can and cannot tell you.

Define the decision, the outcome and the groups

Define the decision, outcome and comparison groups first. A fairness metric answers a question about a specific decision (what the tool decides), a positive outcome (what counts as a correct or good result) and comparison groups (who is compared with whom). Change any of the three and the numbers change meaning.

Invented scenario:

Feature:   ShortlistHelper, the screening feature from page 1, tested on 40 past applicants.
Decision:  shortlisted (1) or not (0). It matters because a person only reads shortlisted CVs.
Positive:  "judged suitable in a later full review of every CV" (this is the label).
Groups:    A and B, placeholders for any comparison attribute that fits your context.
Reference: group A, because it is larger. A convenience, not a ranking of the groups.
Question:  do the tool's decisions and errors fall equally on both groups?

Two choices here are decisions of people, not facts. The positive outcome is a judgement made by reviewers. The reference group is for arithmetic only: every difference is "group minus reference", so the sign tells you the direction. Who defines the groups, and whether you may hold the attribute at all, are questions for your legal or privacy owner; this page only shows the calculation.

Build the toy dataset

Build a small synthetic dataset with group labels. The table is built from counts, so every number can be checked by hand. Each cell says: group, label, prediction, how many people, and how many of them had a career break.

Toy data: 40 invented applicants.

# Toy data: 40 invented past applicants. label = judged suitable in a later full review (1/0),
# pred = shortlisted by the tool (1/0). Group A and B are placeholders for any comparison groups.
CELLS = [  # (group, label, pred, how many people, how many of them had a career break)
    ("A", 1, 1, 9, 0), ("A", 1, 0, 3, 1), ("A", 0, 1, 3, 0), ("A", 0, 0, 9, 3),
    ("B", 1, 1, 3, 1), ("B", 1, 0, 3, 3), ("B", 0, 1, 1, 0), ("B", 0, 0, 9, 6),
]


def build(cells):
    rows = []
    for group, label, pred, n, breaks in cells:
        for k in range(n):
            rows.append({"group": group, "label": label, "pred": pred,
                         "career_break": "yes" if k < breaks else "no"})
    return rows


ROWS = build(CELLS)
print(len(ROWS), "rows")
for g in sorted({r["group"] for r in ROWS}):
    sub = [r for r in ROWS if r["group"] == g]
    print(f"group {g}: {len(sub)} people, {sum(r['label'] for r in sub)} suitable, "
          f"{sum(r['pred'] for r in sub)} shortlisted")
print(ROWS[0], ROWS[-1], sep="\n")

40 rows
group A: 24 people, 12 suitable, 12 shortlisted
group B: 16 people, 6 suitable, 4 shortlisted
{'group': 'A', 'label': 1, 'pred': 1, 'career_break': 'no'}
{'group': 'B', 'label': 0, 'pred': 0, 'career_break': 'no'}

Group A has 12 suitable people and 12 shortlisted. Group B has 6 suitable and 4 shortlisted. Already the sizes are uneven, which matters later.

Protected attributes and proxy variables

Distinguish protected attributes from proxy variables. A protected attribute is a characteristic that you have decided, for ethical or legal reasons, must not drive unfair treatment; which characteristics count depends on place and sector, so ask. A proxy variable is an ordinary-looking input that carries information about one. Deleting the group column does not remove the problem if a proxy stays. Test it: can the group be recovered from the proxy, and does the decision differ by proxy value?

def count(rows, **match):
    return sum(all(r[k] == v for k, v in match.items()) for r in rows)


for g in ("A", "B"):
    print(f"group {g}: career break {count(ROWS, group=g, career_break='yes')} of {count(ROWS, group=g)}")
guess = sum((r["career_break"] == "yes") == (r["group"] == "B") for r in ROWS)
print("group recovered from career_break alone:", guess, "of", len(ROWS))
print("always guessing A would be right for:", count(ROWS, group="A"))
for v in ("yes", "no"):
    n, chosen = count(ROWS, career_break=v), count(ROWS, career_break=v, pred=1)
    print(f"career break {v:<3}: {n} people, {chosen} shortlisted, {count(ROWS, career_break=v, label=1)} suitable")

group A: career break 4 of 24
group B: career break 10 of 16
group recovered from career_break alone: 30 of 40
always guessing A would be right for: 24
career break yes: 14 people, 1 shortlisted, 5 suitable
career break no : 26 people, 15 shortlisted, 13 suitable

Using only career_break, you recover the group correctly for 30 of 40 people; always guessing A gives 24. So the proxy leaks group information. And people with a career break are shortlisted 1 time in 14, against 15 in 26 for the others, even though 5 of the 14 are suitable. That does not prove the tool uses the proxy, or that the proxy is unfair. It tells you to look, and the register risk about career breaks now has a number behind it.

Labels, predictions and observed outcomes

Separate labels, predictions and observed outcomes. Three different things hide behind the word "correct".

label             what we treat as correct         "suitable" in a later review (a human judgement)
prediction        what the tool said               "shortlisted" or not
observed outcome  what happened afterwards         "hired and did well", known only for some people

Each can be wrong in its own way. A label made from past hiring decisions repeats those decisions, fair or not. A prediction depends on a threshold someone chose. An observed outcome exists only for people who were selected, so it cannot show who was wrongly left out. The counts per group are the base of every metric: true positives (tp: suitable and shortlisted), false negatives (fn: suitable, not shortlisted), false positives (fp: not suitable, shortlisted) and true negatives (tn).

Create this small helper file. All later formulas call it. Rates print with two decimals; differences of less than 0.01 are not visible.

# file: fair.py
from collections import Counter

NAMES = {(1, 1): "tp", (1, 0): "fn", (0, 1): "fp", (0, 0): "tn"}   # (label, pred)


def confusion(rows, group=None):
    c = Counter(NAMES[r["label"], r["pred"]] for r in rows if group in (None, r["group"]))
    return {k: c[k] for k in ("tp", "fn", "fp", "tn")}


def ratio(top, bottom):
    return top / bottom if bottom else None      # None means undefined, never 0


def rates(c):
    n, pos, neg, chosen = sum(c.values()), c["tp"] + c["fn"], c["fp"] + c["tn"], c["tp"] + c["fp"]
    return {"n": n, "selection": ratio(chosen, n), "tpr": ratio(c["tp"], pos), "fnr": ratio(c["fn"], pos),
            "fpr": ratio(c["fp"], neg), "accuracy": ratio(c["tp"] + c["tn"], n),
            "precision": ratio(c["tp"], chosen), "recall": ratio(c["tp"], pos)}


def by_group(rows):
    stats = {g: rates(confusion(rows, g)) for g in sorted({r["group"] for r in rows})}
    stats["all"] = rates(confusion(rows))
    return stats


def fmt(x, signed=False):
    return "n/a" if x is None else f"{x:+.2f}" if signed else f"{x:.2f}"


def show(stats, keys):
    print(f"{'group':<7}" + "".join(f"{k:>11}" for k in keys))
    for g, s in stats.items():
        print(f"{g:<7}" + "".join(f"{s[k] if k == 'n' else fmt(s[k]):>11}" for k in keys))


def diff(stats, key, ref):
    """Each group minus the reference group; None when either side is undefined."""
    return {g: None if None in (s[key], stats[ref][key]) else s[key] - stats[ref][key]
            for g, s in stats.items() if g not in (ref, "all")}

Note ratio: when a rate has nothing to divide by, the answer is None, shown as n/a. A group with no suitable people has an undefined true-positive rate, not a rate of zero.

from fair import confusion

for g in ("A", "B", None):
    print(g or "all", confusion(ROWS, g))
shown = [r for r in ROWS if r["pred"] == 1]
print("outcome seen only for the shortlisted:", len(shown), "of", len(ROWS))
print("suitable people the tool missed, invisible to an outcomes-only audit:",
      sum(r["label"] == 1 and r["pred"] == 0 for r in ROWS))

A {'tp': 9, 'fn': 3, 'fp': 3, 'tn': 9}
B {'tp': 3, 'fn': 3, 'fp': 1, 'tn': 9}
all {'tp': 12, 'fn': 6, 'fp': 4, 'tn': 18}
outcome seen only for the shortlisted: 16 of 40
suitable people the tool missed, invisible to an outcomes-only audit: 6

The audit that sees only outcomes would watch 16 people and never see the 6 suitable people the tool missed.

Selection rates

Compute group-level selection rates. The selection rate is the share of a group that the tool selects. It ignores the label completely.

from fair import by_group, show

stats = by_group(ROWS)
show(stats, ["n", "selection"])

group            n  selection
A               24       0.50
B               16       0.25
all             40       0.40

Group A is selected at 0.50, group B at 0.25. The "all" row (0.40) is an average that hides this.

False-positive and false-negative rates

Compute false-positive and false-negative rates. The false-positive rate (fpr) is the share of truly unsuitable people who are shortlisted; the false-negative rate (fnr) is the share of truly suitable people who are not. Which error hurts depends on the decision. Here a false negative is a qualified person who is never read, which harms the applicant. A false positive costs a recruiter some time. In a fraud check it would be the other way round.

show(stats, ["fpr", "fnr"])

group          fpr        fnr
A             0.25       0.25
B             0.10       0.50
all           0.18       0.33

Qualified applicants in B are missed twice as often as in A (0.50 against 0.25), while unsuitable applicants in B are shortlisted less often (0.10 against 0.25). The two errors move in different directions, and a single "error rate" would hide both.

Accuracy, precision and recall

Compare accuracy across groups. Accuracy is the share of correct decisions.

show(stats, ["accuracy"])

group     accuracy
A             0.75
B             0.75
all           0.75

Both groups score 0.75, yet the tool treats them differently, as the previous tables show. Equal accuracy does not mean equal treatment: B has many true negatives that raise its score.

Compute precision and recall by group. Precision is the share of shortlisted people who are suitable. Recall is the share of suitable people who are shortlisted; it is the same number as the true-positive rate.

show(stats, ["precision", "recall"])

group    precision     recall
A             0.75       0.75
B             0.75       0.50
all           0.75       0.67

Precision is 0.75 for both groups, so a recruiter who reads the shortlist sees the same share of suitable people in either. Recall differs: half of B's suitable people reach the shortlist, three quarters of A's.

Three fairness differences

Hardt, Price and Srebro (2016) define two criteria in terms of the real outcome. In words:

Equal opportunity is a relaxation of equalised odds that requires only the first half. Examine the demographic-parity difference by comparing selection rates, and the equal-opportunity and equalised-odds differences by comparing the rates above. For equalised odds this page reports the largest of the two gaps. The tolerance is your choice and needs a written reason.

from fair import diff, fmt

TOL = 0.05   # a tolerance is a choice; no value here is a standard
dp, tpr, fpr = (diff(stats, k, "A") for k in ("selection", "tpr", "fpr"))
for g in dp:
    worst = max(abs(tpr[g]), abs(fpr[g]))
    print(f"demographic parity difference ({g} - A): {fmt(dp[g], True)}  within {TOL}: {abs(dp[g]) <= TOL}")
    print(f"equal opportunity difference ({g} - A):  {fmt(tpr[g], True)}  within {TOL}: {abs(tpr[g]) <= TOL}")
    print(f"false-positive-rate difference:        {fmt(fpr[g], True)}")
    print(f"equalised odds, largest gap:           {fmt(worst)}  within {TOL}: {worst <= TOL}")

demographic parity difference (B - A): -0.25  within 0.05: False
equal opportunity difference (B - A):  -0.25  within 0.05: False
false-positive-rate difference:        -0.15
equalised odds, largest gap:           0.25  within 0.05: False

All three show a gap of 0.25 or more against a tolerance of 0.05. Notice that the fpr gap (-0.15) favours group A's unsuitable applicants, while the tpr gap disadvantages B's suitable ones. A gap of zero on one metric is possible while another stays large (the Medium exercise shows it).

Which criterion should a team prefer? Reasonable people disagree, and the strongest arguments are these.

The choice depends on the harm in the decision, the quality of the label and the context, and teams in different sectors and places choose differently. The page does not pick one.

What these numbers cannot tell you

A responsible finding therefore reads: "On this table, for this decision and label, group B's suitable applicants were shortlisted at 0.50 against 0.75; the group is small, the label is unreviewed, and we propose further checks." It does not read "the tool is biased" or "the tool is fair".

Where this shows up at work

Analysts run a group table like this before a pilot and after each change, then take it to the people who own the decision. The same code with real data would also need a privacy review for the group column.

Common mistakes

1. Dropping the group column and assuming the problem is gone. A proxy can carry the same information. Check the proxy as shown above.
2. Reading accuracy as fairness. Equal accuracy hid a doubled miss rate for B. Always show the error rates by group.
3. Averaging groups together. The "all" row hid a 2-to-1 difference in selection.
4. Turning an undefined rate into zero. A group with no positives has no tpr. Report n/a and do not compare.
5. Treating the label as truth. Say who made it and how, and what it leaves out.

Try it yourself

Use the namespace from this page. The exercises revisit false positives and false negatives from Fairness and Its Trade-Offs: Inclusion, Accessibility and Language in the Beginner tutorial.

1. Easy. For group B (tp 3, fn 3, fp 1, tn 9), work out by hand its selection rate, fpr and precision. Then check against the tables.
2. Medium. Rebuild group B so that its true-positive rate equals A's (0.75) but it is selected much less often. Print selection, tpr and fpr and both gaps. Which metrics are satisfied?
3. Hard. Add a group C of 3 people, none suitable, 1 shortlisted. Print its row. Explain why tpr is n/a and what you would do with group C in a report.

Answers

Easy. Selected: 3 + 1 = 4 of 16, so 0.25. Unsuitable: 1 + 9 = 10, of whom 1 is shortlisted: fpr 0.10. Shortlisted 4, of whom 3 suitable: precision 0.75. Pass checklist: the three values match the tables; you used the right denominator for each (everyone, unsuitable people, shortlisted people). A weak answer divides by the wrong group.

Medium. Suitable people: 3 + 1 = 4, shortlist 3 of them (tpr 0.75); no unsuitable person is shortlisted.

alt = build([("A", 1, 1, 9, 0), ("A", 1, 0, 3, 1), ("A", 0, 1, 3, 0), ("A", 0, 0, 9, 3),
             ("B", 1, 1, 3, 1), ("B", 1, 0, 1, 1), ("B", 0, 0, 12, 7)])
s2 = by_group(alt)
show(s2, ["n", "selection", "tpr", "fpr"])
print("selection gap:", fmt(diff(s2, "selection", "A")["B"], True), "| tpr gap:", fmt(diff(s2, "tpr", "A")["B"], True))

group            n  selection        tpr        fpr
A               24       0.50       0.75       0.25
B               16       0.19       0.75       0.00
all             40       0.38       0.75       0.12
selection gap: -0.31 | tpr gap: +0.00

Equal opportunity is satisfied (gap +0.00), demographic parity is not (-0.31), and the fpr gap is -0.25. Pass checklist: tpr equals 0.75; you report which metrics hold and which do not; you do not call the result fair or unfair. A weak answer reports only the metric that passes.

Hard.

more = build([("A", 1, 1, 9, 0), ("A", 1, 0, 3, 1), ("A", 0, 1, 3, 0), ("A", 0, 0, 9, 3),
              ("C", 0, 1, 1, 0), ("C", 0, 0, 2, 1)])
s3 = by_group(more)
show(s3, ["n", "selection", "tpr", "fpr", "precision"])
print(diff(s3, "tpr", "A"))

group            n  selection        tpr        fpr  precision
A               24       0.50       0.75       0.25       0.75
C                3       0.33        n/a       0.33       0.00
all             27       0.48       0.75       0.27       0.69
{'C': None}

With no suitable people, "the share of suitable people selected" has no denominator, so tpr is undefined and the difference is None. Pass checklist: n/a is kept, not replaced by 0.00; group C is reported with its counts and not compared on tpr or recall; the note says that a group of 3 is too small for any conclusion. A strong answer suggests collecting more data or merging groups only with a stated reason.

Summary

• Define decision, positive outcome and comparison groups first; every metric depends on them.
• Test proxy variables as well as protected attributes; removing a column is not enough.
• Keep labels, predictions and observed outcomes separate; outcomes exist only for selected people.
• Compute selection, error, accuracy, precision and recall per group, and report undefined rates as n/a.
• Demographic parity, equal opportunity and equalised odds can disagree; a gap is a reason to look, not a verdict, and toy data proves nothing about the real world.

Next: Comparing Fairness Goals and Their Limits: Small Samples and Conflicting Metrics shows why these metrics can conflict, how base rates and small groups change the picture, and how to compare two thresholds.