Where Bias Comes From: Data, Design, Proxies and Feedback Loops

An AI tool can treat groups of people differently even when no one meant it to. Nobody has to write a rule that says "treat this group worse". The difference can come from the examples the system learned from, from how success was measured, from who was left out, or from the people who check the result. If you know where to look, you can often notice the problem early, before it affects anyone.

This page is a map of the places where bias can enter. It builds on the decision pattern from Everyday AI Ethics: Who Benefits and Who Could Be Harmed: step 2 asks who is affected, and this page helps you see who is affected unevenly. In this course, bias means a systematic difference in how a system treats or serves different groups. The word does not mean that a person was prejudiced, and it does not by itself say whether a difference is unfair. That judgement comes later, on the next page.

What you will learn

• The difference between bias in data and bias in a decision, and the main places bias can enter: history, who is represented, measurement and proxies, selection, and human reviewers.
• How a system's own outputs can feed back into its future data and make a difference grow.
• How language, dialect, names, addresses and cultural references can quietly change results, and what accessibility barriers look like.
• Why one overall score can look good while one group is served badly, with a small worked table.
• A short checklist for finding three possible sources of bias in a sample decision.

Bias in data and bias in decisions

Data is a record of what happened or what someone wrote down. A decision is what the system, or a person using the system, does with it. The two are linked but not the same, and you need to check both.

Two cautions. First, biased data does not always produce a biased decision: a team can notice and correct it. Second, fair-looking data can still lead to an unfair decision, for example when the decision rule uses a feature that works as a stand-in for a group. The next sections show how.

Where bias can enter

Researchers have described several sources of harm across the life of a machine-learning system. One research paper often cited for this is a 2019 framework by Suresh and Guttag (a paper on the arXiv preprint server). It names seven sources: historical, representation, measurement, aggregation, learning, evaluation and deployment bias. This page uses the ones a beginner can spot in everyday work, plus two common terms that the paper does not use as headings: selection bias (a common name for a kind of unrepresentative sample, close to the paper's representation bias) and human reviewer bias. It also adds a plain description of feedback loops. The groupings here are our teaching choices, not a standard.

The world       ->  Data collected  ->  Labels and  ->  Model or   ->  Human      ->  Decision
(history,           (who is in it,      measures        rules         reviewer        and
 unequal access)     who is missing)    (proxies)                     (checks it)     outcome
        ^                                                                                |
        +----------------------- outcomes become tomorrow's data -----------------------+

Use the diagram as a checklist. For each arrow, ask: could a difference between groups be introduced or made larger here?

Invented scenario:

Nadia manages admissions for a free evening skills course at a community college in Dhaka.
About 1,000 people apply each year. A tool reads each short application and marks it
"Invite to interview" or "Not now". It was set up using five years of past admissions.
Nadia's team reads every "Invite" and about one in ten "Not now" applications.

Historical bias

The past records may reflect old unfairness, even if they were collected correctly. If, in the past, the course mostly admitted people from a few neighbourhoods because of where it advertised, the tool learns "good applicant" from that pattern. Nothing is wrong with the arithmetic. The world it learned from was uneven. Question to ask: would a perfect copy of the past be what we want in the future?

Representation bias

Some people are missing, or appear only a little, in the data. If few past applications came from older learners or from people writing in a second language, the tool has little to learn from for them. The usual result is more mistakes for the groups it saw least. Question to ask: who is in the data, and who should be but is not?

Measurement bias: labels and proxies

Systems learn from labels, such as "was admitted" or "finished the course". A label is a stand-in for what you really care about, which may be hard to measure. "Finished the course" may measure free time and stable transport as much as ability. A proxy is a feature that stands in for something else. A postal code can stand in for neighbourhood, income, or community. A gap in work history can stand in for caring duties, illness or study, and the tool cannot tell the difference. Question to ask: what are we really measuring, and for whom is it a poor measure?

Selection bias

Who got into the data in the first place? If only people who applied online are recorded, people who could not use the website are invisible. If only successful graduates are followed up, the stories of those who left early are never heard. Question to ask: what had to be true for a person to appear in this dataset?

Bias introduced by human reviewers

Human review is meant to be a safeguard, but reviewers have their own habits. They may agree too easily with an AI suggestion, or favour applications that look familiar. A reviewer who sees only the "Not now" cases written in a certain style may start to expect them. Nadia's rule to read every "Invite" but only one in ten "Not now" means errors in the "Not now" group are much less likely to be caught. Question to ask: which mistakes can the reviewer actually see?

Feedback loops

Sometimes a system's own outputs become tomorrow's data. If the tool rarely invites people from one background, fewer of them join the course. Next year's "successful past admissions" then contain even fewer of them, and the tool learns that they are less likely to succeed. The gap grows without any new decision to make it grow. Question to ask: does what the system does today change the data it learns from tomorrow?

Where reasonable people disagree

Should a proxy that predicts well be used if it also works as a stand-in for a group?

Both views want good decisions. They weigh accuracy against the risk of repeating a disadvantage differently, and they may be right in different settings. A useful middle step is to test with and without the feature and look at results by group, then decide and write down why.

Language, dialect and regional bias

Text is not neutral. Models learn mostly from the language they see most, so they can work better for some languages, spellings and writing styles than for others.

• Language and dialect. Two people may write the same meaning in different varieties of the same language. A research paper (Hofmann and colleagues, 2024, on arXiv) reported that several language models held more negative hidden stereotypes about speakers of African American English than recorded human stereotypes, and in hypothetical decisions assigned them less prestigious jobs and harsher outcomes based only on dialect. Training with human feedback did not remove it. The finding concerns the specific models tested. The lesson is general: bias can sit in how something is written, not only in what it says.
• Cost and quality across languages. A 2023 paper (Ahia and colleagues, on arXiv) found that the same content can use different numbers of tokens in different languages, so some speakers can pay more and receive worse results from the same service. You met tokens in How AI Works: LLMs, Tokens and Limitations.
• Names. A tool that has seen few examples of a name may misspell it, split it oddly, or treat it as less likely to be "real". Names such as Nguyen Thi Lan, Ōtsuka Aiko or Kwame Mensah-Boateng show how different naming orders and spellings can be.
• Addresses. Many places do not use "street number, street name". A Dhaka address may read "বাজারের পাশে, মসজিদ গলি" (beside the market, mosque lane). A form that requires a five-digit postal code or a street name can reject a real address.
• Cultural references. A prompt or question that assumes a particular holiday, school system or sport may be clear to one group and confusing to another.

A simple test is a paired comparison: take one real task, change only the name, language variety or address style, and compare what the system returns. Keep the content identical. If results differ, write down how.

Accessibility barriers

A system can be biased in practice if some people cannot use it at all. Accessibility is a design question, not a medical one. Here are everyday barriers to look for.

We cite no study here. Treat the table as a set of design questions you can ask yourself. Asking the people who use a service is better than guessing about them.

Why an average score hides differences

An overall number answers "how is the system doing for everyone together?" It does not answer "how is it doing for each group?" When one group is much smaller than another, its experience hardly moves the average.

Toy data:

Nadia's tool, checked against 1,000 past applications that people have reviewed.
"Correct" means the tool's mark matched the careful human decision.

Group                                   Applications   Correct   Correct share   Wrong
A: written in the course's main language      900         864        96.0%          36
B: written in a second language               100          70        70.0%          30
All together                                1,000         934        93.4%          66

The overall score is 93.4%, which sounds strong. Group B's score is 70.0%, so about three in ten of its applications are marked wrongly, compared with about one in twenty-five for Group A. Group B is only 10% of applications but accounts for 30 of the 66 errors, about 45%. (We worked these figures out with a short calculation; the data is invented, so the example only shows the arithmetic and says nothing about any real tool.)

Two more habits follow. Look at counts as well as percentages, because a small group's rate can jump a lot with one case: in a group of 10, one extra correct case moves the score by ten points. And if a group is very small, say so and avoid strong claims either way. Also, if you cannot measure a group at all, because nobody recorded it, remember that "no difference found" means "not measured".

Where reasonable people disagree

To compare groups you usually have to record who belongs to which group. Is that wise?

Views differ across countries and communities, and the rules about collecting such information differ by country and sector. Check the current official rules and your organisation's policy before you decide. Some teams use voluntary self-description, a sample review, or an outside reviewer instead of collecting everything.

Documented case: Gender Shades

Documented case:

Gender Shades, 2018. Buolamwini and Gebru, Conference on Fairness, Accountability
and Transparency (published in PMLR volume 81).
Established: they tested three commercial gender-classification systems. Error rates were
much higher for darker-skinned women than for other groups; the paper's abstract reports
up to 34.7% for darker-skinned women against at most 0.8% for lighter-skinned men. They
also report that two widely used benchmark datasets were mostly lighter-skinned subjects
(79.6% in one, 86.2% in the other).
Not covered: later versions of these products, and other kinds of face tasks.

What it teaches: the benchmark data under-represented some groups, which is representation bias, and the errors fell unevenly between groups. This is the same pattern as Nadia's table, found in real products at a particular time. Separate work by a United States government laboratory (NIST, 2019) evaluated face-recognition algorithms and found that error rates differ by demographic group and between algorithms. Both studies concern their specific systems and dates. They do not show that every system behaves the same way, and they do not tell you what to conclude about any product you use today.

Finding bias: a short checklist

Template:

Bias check for one decision
1. What is decided, and about whom?
2. Which groups might be served differently? (language, age, region, disability, other)
3. HISTORY:        does the data copy an unequal past?
4. REPRESENTATION: who is missing or rare in the data?
5. MEASURES:       what do the labels and features really stand for? Any proxies?
6. SELECTION:      who could not get into the data at all?
7. REVIEWER:       which errors can the human reviewer see? Do they follow the tool too easily?
8. FEEDBACK:       will today's results change tomorrow's data?
9. NUMBERS:        results for each group, with counts. Any group too small to judge?
10. ACCESS:        can everyone actually use it? What is the alternative?

Where this shows up at work

A support team notices that the chatbot handles polite, formal messages well and struggles with short, informal ones. A university uses a language tool and finds that non-native writers are corrected more. A small business uses an AI scheduling helper that cannot read a customer's regional address format. In each case, the person who asked "for whom does this work less well?" found the problem sooner and could fix it before it caused harm.

Common mistakes

• Trusting the overall score. It can hide a group with poor results. Fix: always ask for results by group and the number of cases in each.
• Assuming "no group information" means "no bias". Bias can act through proxies and language. Fix: check the features and run a paired comparison.
• Blaming only the data. The labels, the decision rule, the reviewer and the feedback loop can each add or increase a difference. Fix: walk through every arrow in the diagram.
• Treating a proxy as neutral because it looks neutral. A postal code is not "just a number". Fix: ask what it could stand in for.
• Forgetting who cannot use the tool. Fix: ask about access before launch and offer another route.

Try it yourself

Use this sample decision for all three tasks. Invented scenario: Haruka coordinates a school in Osaka. A tool marks each pupil "Offer extra reading support" or "No support now". It was trained on the last four years of pupils who were offered support and then improved. It uses the home postal area, the number of books at home from a parent survey, and an online reading test that pupils complete at home. Teachers read only the "Offer" list.

1. Easy. From the sample, name three possible sources of bias, each matched to a name from this page.
2. Medium. For each of the three, write what you would check first and what evidence would show whether it is a real problem.
3. Hard. Haruka's check finds 92% agreement between the tool and a careful human review of 500 pupils. Explain in four to six sentences why that number is not enough. Name two pieces of information she needs, and say how she could collect them. Then write one condition she should set before keeping the tool, using the decision pattern from page 1.

Answers

Easy. A strong answer picks three that fit: (1) Measurement and proxy: the number of books at home can stand in for family income or culture, not for a pupil's need. (2) Measurement and access: an online test done at home measures a quiet place and a working device as well as reading. (3) Historical or feedback loop: learning from pupils who were offered support in the past may repeat who was offered it before, and pupils who get "No support now" never appear as improved cases next year. Other valid choices: human reviewer (only the "Offer" list is read, so missed pupils are never seen), selection (parents who did not return the survey are missing) and representation (few pupils learning in a second language in the history). Pass: three different sources, each tied to a detail in the sample. Weak: three answers that name the same source, or a generic "the AI might be biased".

Medium. Model answers: for the survey, test the tool with that feature removed and compare results by group; look at how many pupils have no survey answer. For the home test, compare scores of pupils who took it at home with a short supervised test at school. For the history, count who was offered support in each area and language group over four years and ask whether that is what the school wants to repeat. For the reviewer problem, read a random sample of "No support now" cases, not only the "Offer" list. Pass: each check is concrete and could change a decision. Weak: "look into it" with no evidence named.

Hard. A strong answer: 92% overall can coexist with a much lower rate for a smaller group, as in the toy table (93.4% overall and 70.0% for Group B). She needs results split by group (such as home language, survey answered or not, and where the test was taken) with counts in each, and the kinds of mistakes: pupils who needed support but were missed, and pupils offered support who did not need it. She can get this by adding group information to the 500 reviewed cases where it is appropriate and permitted, or by sampling from each group, and she should check her school's rules on recording such data. One condition: "Keep the tool only if, at the next review, no group with at least 30 pupils has an agreement rate more than 10 points below the overall rate; a teacher reads 20 'No support now' cases every month; review on 2026-12-15; Haruka is accountable." The numbers are examples, not a standard. Pass: explains why an average hides a group, names two needed pieces of information, gives a way to collect them, and sets a condition with an owner and a date. Weak: calls 92% "good enough" or says the tool must be dropped without evidence.

Look back at the worksheet on page 1. Haruka's problem fits step 3 (benefits and harms), step 5 (evidence) and step 6 (decision with conditions).

Summary

• Bias means a systematic difference in how groups are served. It can enter through history, who is represented, labels and proxies, who got into the data, human reviewers and feedback loops.
• Language, dialect, names, addresses and cultural references can change results quietly, so compare results with one thing changed at a time.
• Accessibility is part of fairness: a tool that some people cannot use serves them badly. Ask who is shut out and what the alternative is.
• An overall score can hide a poor result for a smaller group. Ask for results by group, with counts.
• A documented case such as Gender Shades shows these patterns in real products at a particular time, without showing that every product behaves the same way.

Next: Fairness and Its Trade-Offs: Inclusion, Accessibility and Language looks at what "fair" can mean once you have found a difference, and why different fairness goals can pull in different directions.