Data Governance: Purpose, Consent, Minimisation, Quality and Re-Identification

The Beginner pages Privacy, Consent and Data Use: What You Share with an AI Tool and Data Rights, Retention and Privacy Practice: Habits That Protect People gave you habits for one person's use of one tool. A team needs records: which datasets exist, who owns them, what each field is for, and what each may be used for next. This page starts with the running example, the screening feature from page 1, and builds those records with Python that reports what is missing instead of guessing.

You need Python 3.12, standard library only. Type the blocks in order into one session; they share variables, and the text under a block is its real output.

Honesty convention. Every dataset, company, role and number is invented and labelled Toy data:. Results show how a calculation works. They are not evidence about any real system, and nothing here is legal advice. Ethical concern, legal obligation and company policy are different things; this page marks which is which.

What you will learn

• How to inventory data sources and owners, record the purpose of each field, and have a program report missing entries.
• How to tell the authority to collect data from permission to reuse it, and what to do when permission is missing or unclear.
• How to record provenance and transformations, and how to count re-identification risk in a de-identified table.
• How to judge consent quality and power imbalances, write retention and deletion rules, and map data access to job responsibilities.
• How to list the transfers of data across organisational boundaries.

Inventory: sources, owners and permissions

Inventory data sources and their owners. A data inventory lists each dataset and answers the same questions every time: where it came from, who is accountable for it (a named role, not "the team"), why it is held, under what permission it was collected, whether it may be reused for the new feature, where it has been, and how long it is kept. The rule for the program: an empty answer is reported as MISSING, and a vague answer ("not stated", "unknown") as UNCLEAR. It never fills a gap.

Toy data: four invented datasets held by Harbourline Staffing for ShortlistHelper.

NEED = ["source", "owner", "purpose", "collected_under", "reuse_ok", "provenance", "retention"]
UNCLEAR = {"", "not stated", "unknown", "tbc"}
INVENTORY = {
    "D1 applicant CVs": {"source": "applicants upload", "owner": "Head of recruitment",
        "purpose": "assess applicants for one job", "collected_under": "application notice v3",
        "reuse_ok": "not stated", "provenance": "upload, then text extracted by vendor tool",
        "retention": "12 months after the job closes"},
    "D2 past hiring outcomes": {"source": "agency records", "owner": "Head of recruitment",
        "purpose": "label for scoring tests", "collected_under": "employment records",
        "provenance": "exported 2026-02"},
    "D3 job requirements": {"source": "hiring managers", "owner": "Head of recruitment",
        "purpose": "tell the feature what a job needs", "collected_under": "internal work product",
        "reuse_ok": "yes, internal", "provenance": "typed by hiring manager", "retention": "3 years"},
    "D4 public profiles": {"source": "public web", "purpose": "extra signals",
        "collected_under": "public web", "reuse_ok": "unknown"},
}


def review(inventory):
    for name, rec in inventory.items():
        missing = [f for f in NEED if f not in rec]
        unclear = [f for f in NEED if f in rec and rec[f] in UNCLEAR]
        print(f"{name:<25} MISSING: {', '.join(missing) or '-'} | UNCLEAR: {', '.join(unclear) or '-'}")


review(INVENTORY)

D1 applicant CVs          MISSING: - | UNCLEAR: reuse_ok
D2 past hiring outcomes   MISSING: reuse_ok, retention | UNCLEAR: -
D3 job requirements       MISSING: - | UNCLEAR: -
D4 public profiles        MISSING: owner, provenance, retention | UNCLEAR: reuse_ok

Only D3 is complete. D1 has an owner and a purpose, but nobody has written down whether its CVs may be reused to test a scoring feature. D2 lacks a reuse answer and a retention rule. D4 has no owner, no provenance and no retention, and its permission is "unknown": being public is a fact about access, not an answer about permission (the Beginner page on privacy and data use made this point in words).

Record the purpose for each data field. The same idea works one level down. If no one can say what a field is for, the safest question is whether to hold it at all.

FIELDS = {"name": "contact the applicant after shortlisting", "email": "contact the applicant after shortlisting",
          "cv_text": "the fit score for one job", "postcode": "estimate the commute",
          "date_of_birth": "", "photo": ""}
for field, purpose in FIELDS.items():
    print(f"{field:<14}", purpose or "MISSING: remove it, or write the purpose and who approved it")

name           contact the applicant after shortlisting
email          contact the applicant after shortlisting
cv_text        the fit score for one job
postcode       estimate the commute
date_of_birth  MISSING: remove it, or write the purpose and who approved it
photo          MISSING: remove it, or write the purpose and who approved it

Two fields have no purpose. A third, postcode, has one, but page 5 showed how an ordinary-looking input can act as a proxy for a protected attribute, so a purpose does not end the review. Writing it down is what lets the review happen.

Collection authority and permission for reuse

Distinguish collection authority from permission for reuse. Collection authority is the permission or justification under which data was first collected, for the purpose people were told. Permission for reuse is a separate question: does that permission cover this new use? Having the data is not an answer.

Invented scenario:

Situation:  Harbourline wants to test ShortlistHelper on last year's CVs (D1).
Collected:  under "application notice v3": "we use your CV to assess your application".
New use:    scoring tests for a tool. The notice does not mention tools or testing.
Question:   does the original permission cover the new use? Nobody has decided.
Options:    A use D1 anyway;  B ask applicants for a separate, optional permission;
            C use only CVs collected after a new notice;  D test on synthetic CVs, not D1.
Unknowns:   how the privacy owner reads the notice; what applicants would reasonably expect.

Reasonable people weigh this differently. Option A is fastest and may be defensible if the notice is broad and the privacy owner agrees; but applicants did not expect it, and a mistake affects many people. B respects choice but gives a smaller, self-selected sample, which can skew the test. C is clean but slow. D costs realism. The decision pattern asks you to choose, name conditions and an accountable owner: for example "D now, C within six months, privacy owner decides whether A is ever acceptable". The legal reading of a notice belongs to your privacy or legal owner; the inventory's job is to make the question visible.

Provenance and transformations

Document data provenance and transformations. Provenance is the history of a dataset: where it began and what was done to it. A transformation is any change: extracting text, removing names, merging, filtering, labelling. Each step can lose information, add error or change the privacy status, so each is recorded with a date and a person or tool.

Template:

Provenance record: D1 applicant CVs
step  date        who or what          action                         effect to note
1     2026-03-01  applicants           uploaded CV files              original, with names
2     2026-03-02  vendor text tool     extracted plain text           tables and layout lost
3     2026-03-05  recruitment analyst  replaced names with codes      pseudonymised; key held by
                                                                      Head of recruitment
4     2026-03-06  analyst              dropped CVs under 50 words     sample now skews to long CVs
Unknown: how the vendor tool treats non-Latin scripts. Owner of this record: analytics lead.

Step 4 is easy to forget and matters later: a filter shapes who is in the data, and so what any fairness table (page 6) can say. Step 3 is pseudonymisation, discussed below: the codes are still personal data while the key exists.

Missing or unclear permissions

Identify missing or unclear data permissions. The inventory output already shows the cases. A workable response ladder, from most to least cautious:

Re-identification in de-identified data

Assess re-identification risk in de-identified data. Removing names does not make a table anonymous. Quasi-identifiers are ordinary attributes (area, birth year, job) that are not unique alone but can single a person out together, especially when someone holds another list that contains the same attributes with names. A simple check counts, for each combination, how many records share it. A k-anonymity-style count asks: what is the smallest group size k, and how many records sit alone in their group (k = 1)?

Toy data: 40 invented people with district, birth year, role and a private outcome, made with a fixed seed.

import random
from collections import Counter

rng = random.Random(42)
RECORDS = [{"district": rng.choice(["D1", "D2", "D3", "D4", "D5"]), "birth_year": rng.randint(1980, 1999),
            "role": rng.choice(["analyst", "nurse", "driver", "teacher"]),
            "outcome": rng.choice(["yes", "no"])} for _ in range(40)]
REGION = {"D1": "R1", "D2": "R1", "D3": "R2", "D4": "R2", "D5": "R2"}
LEVELS = {"exact": lambda r: (r["district"], r["birth_year"], r["role"]),
          "coarse": lambda r: (REGION[r["district"]], r["birth_year"] // 10 * 10, r["role"]),
          "coarser": lambda r: (REGION[r["district"]], r["role"])}

for name, key in LEVELS.items():
    sizes = Counter(key(r) for r in RECORDS)
    alone = sum(1 for v in sizes.values() if v == 1)
    print(f"{name:<8} groups {len(sizes):>2} | records alone {alone:>2} | smallest group k = {min(sizes.values())}")

exact    groups 40 | records alone 40 | smallest group k = 1
coarse   groups 15 | records alone  4 | smallest group k = 1
coarser  groups  8 | records alone  1 | smallest group k = 1

In the exact table all 40 people are alone in their group. Generalising (district to region, birth year to decade) leaves 4 alone; generalising further leaves 1. Now a linkage test: a second list, with names, from some other source (invented too). How many people does each level single out, and what would the private outcome reveal?

PUBLIC = [("Ana", RECORDS[3]), ("Bo", RECORDS[8]), ("Cy", RECORDS[15]), ("Di", RECORDS[22]),
          ("Eli", RECORDS[30]), ("Fay", {"district": "D2", "birth_year": 1990, "role": "nurse"})]  # Fay is not in the table
for name, key in LEVELS.items():
    learned = []
    for person, attrs in PUBLIC:
        match = [r for r in RECORDS if key(r) == key(attrs)]
        if len(match) == 1:
            learned.append(f"{person}={match[0]['outcome']}")
    print(f"{name:<8} singled out {len(learned)} of {len(PUBLIC)}:", ", ".join(learned) or "none")

exact    singled out 5 of 6: Ana=yes, Bo=no, Cy=no, Di=no, Eli=no
coarse   singled out 1 of 6: Fay=yes
coarser  singled out 0 of 6: none

At the exact level five of six named people are singled out and their private outcome is read off. At the coarse level one is matched, and that match is wrong: Fay is not in the table at all. A unique match on coarse data may be a different person, which is part of how generalising protects people. The limits of this check are important. It uses only the attributes you chose; an outsider may know more. It says nothing about the sensitive column itself: if every member of a group shares the same outcome, the outcome leaks even at k = 5. It is a screening count, not a guarantee and not a legal test.

Documented case:

Year:        2008 (IEEE Symposium on Security and Privacy; arXiv cs/0610105).
Established: Narayanan and Shmatikov studied the Netflix Prize dataset, about 500,000
             subscribers, from which names had been removed. With limited background
             knowledge about a subscriber, the authors could locate that person's record.
             Using the public IMDb site as background knowledge, they matched known
             users to records and uncovered apparent political preferences.
Lesson:      removing names is not enough when a record is unique or nearly unique.

Legal note (as of 2026-10-12; not legal advice): Under the GDPR, in plain words from an unofficial text site (the official text is on EUR-Lex): Recital 26 says that pseudonymised data is still personal data. To decide whether a person is identifiable, account should be taken of all the means reasonably likely to be used, including cost, time and technology. Truly anonymous data is outside the Regulation. Whether a given table counts as anonymous is a legal and technical judgement, not something a count of k decides. Source type: the Recital text. Check the current official text for your country and sector.

Consent quality and power imbalances

Evaluate consent quality and power imbalances. A tick in a box is a record of a click. Whether it is good consent depends on the conditions around it. The questions below describe good practice; whether consent is legally valid is a separate question for your privacy owner.

A power imbalance exists when one side controls something the other needs: an applicant needs the job, an employee needs the pay, a student needs the grade, a patient needs the care. Where the imbalance is large, a "yes" may be given because refusing feels costly, whatever the form says. Reasonable people disagree about what follows. Some hold that consent should rarely be the basis in such settings and that notice and necessity should do the work; others hold that well-designed optional choices still respect people. Either way the practical question is the same: could this person say no without harm, and would we notice if they did?

Retention and deletion rules

Define retention and deletion rules. Retention is a design principle: keep only what the stated purpose needs, for a defined time, with a deletion route. Your law or sector may set limits too; ask. A rule covers more than the main dataset.

The weak version is "we keep data as long as it is useful": useful for whom, and who checks? Page 8 builds a retention schedule in Python.

Mapping data access to job responsibilities

Map data access to job responsibilities. People should reach the fields their job needs and no more. Write the need first, then compare it with what is actually granted.

Toy data: five roles at the invented agency.

NEEDED = {"recruiter": {"name", "email", "cv_text", "score"}, "hiring manager": {"cv_text", "score"},
          "analyst": {"score", "outcome_label"}, "vendor support": set(), "IT admin": set()}
GRANTED = {"recruiter": {"name", "email", "cv_text", "score", "date_of_birth"},
           "hiring manager": {"name", "email", "cv_text", "score"},
           "analyst": {"score", "outcome_label", "name", "cv_text"}, "vendor support": {"cv_text"},
           "IT admin": {"name", "email", "cv_text", "score", "date_of_birth", "outcome_label"}}
for role in NEEDED:
    extra, lacking = sorted(GRANTED[role] - NEEDED[role]), sorted(NEEDED[role] - GRANTED[role])
    print(f"{role:<15} no stated need: {', '.join(extra) or '-'} | needed but not granted: {', '.join(lacking) or '-'}")

recruiter       no stated need: date_of_birth | needed but not granted: -
hiring manager  no stated need: email, name | needed but not granted: -
analyst         no stated need: cv_text, name | needed but not granted: -
vendor support  no stated need: cv_text | needed but not granted: -
IT admin        no stated need: cv_text, date_of_birth, email, name, outcome_label, score | needed but not granted: -

The result is a list of questions, not a verdict. The recruiter's date of birth has no stated need. The analyst can see names although the task needs only scores and labels. Vendor support and IT admin have an empty need, so every field they reach is a question: an administrator may legitimately need broad access to run the system, but then the need is "maintenance, logged, time-limited", and that is written down and reviewed.

Transfers across organisational boundaries

Identify data transfers across organisational boundaries. Each time data leaves the organisation that holds it, a new party can see, keep or reuse it. List every flow with what moves, why, under what written agreement, and what happens to the data afterwards.

FLOWS = [("applicants", "Harbourline", "CV, contact details", "application notice v3", "12 months"),
         ("Harbourline", "ScoreVendor (invented)", "CV text for scoring", "MISSING", "MISSING"),
         ("Harbourline", "client employer", "shortlist of 5 CVs", "client contract clause 7", "client deletes after hire")]
for src, dst, what, agreement, afterwards in FLOWS:
    flag = "  <-- ask before continuing" if "MISSING" in (agreement, afterwards) else ""
    print(f"{src} -> {dst}: {what} | agreement: {agreement} | afterwards: {afterwards}{flag}")

applicants -> Harbourline: CV, contact details | agreement: application notice v3 | afterwards: 12 months
Harbourline -> ScoreVendor (invented): CV text for scoring | agreement: MISSING | afterwards: MISSING  <-- ask before continuing
Harbourline -> client employer: shortlist of 5 CVs | agreement: client contract clause 7 | afterwards: client deletes after hire

The vendor flow has no agreement and no end-of-use rule recorded. Perhaps they exist and were never filed; the register's job is to find out. Page 8 looks at where the vendor itself sends data.

Where this shows up at work

Before a pilot, a privacy or governance reviewer asks for the inventory, the provenance record and the access list. A team that can produce them in minutes, with gaps stated honestly, usually gets a faster and more specific decision than one that answers from memory.

Common mistakes

1. "We already have the data." Holding data is not permission to use it for a new purpose. Record collection authority and reuse separately.
2. Treating public data as free to use. Public describes access. Check owner, terms and permission, and record them.
3. Calling a table anonymous because names are gone. Count unique combinations and test linkage.
4. A consent box as the whole story. Check informed, specific, free and reversible, and ask about power.
5. Filling blanks with guesses. A record that says MISSING is useful; one that invents an owner hides the problem.

Try it yourself

Use the namespace from this page. The exercises reuse the running example and the proxy idea from page 5.

1. Easy. From the inventory output, list which datasets you would not use for ShortlistHelper tomorrow, and the one question that would unblock each.
2. Medium. Add a fifth dataset, "D5 interview notes", with a source, owner, purpose and retention but no collected_under and a reuse_ok of "tbc". Run review. What does it report, and what is the first action?
3. Hard. A vendor-support person asks for read access to cv_text to debug a failed upload. Write the access decision using the decision pattern: need, options (including no access), conditions, owner and review date.

Answers

Easy. D4 (no owner, provenance or retention; permission unknown): "Who owns it, and do the source's terms permit this use?" D1 (reuse not stated): "Does the privacy owner read notice v3 as covering scoring tests?" D2 (no reuse answer, no retention): "May outcome records be used as labels, and for how long?" D3 can be used. Pass checklist: D3 is the only dataset cleared; each blocked dataset has one concrete question and an owner to ask; the answer does not assume that public or internal data needs no permission. A weak answer clears D1 because "we collected it ourselves".

Medium.

INVENTORY["D5 interview notes"] = {"source": "recruiters", "owner": "Head of recruitment",
    "purpose": "record interview impressions", "reuse_ok": "tbc", "provenance": "typed after interviews",
    "retention": "6 months"}
review({"D5 interview notes": INVENTORY["D5 interview notes"]})

D5 interview notes        MISSING: collected_under | UNCLEAR: reuse_ok

It reports collected_under as MISSING and reuse_ok as UNCLEAR. The first action is to ask the owner under what notice the notes are written and whether interviewees know they exist; do not use the notes for the feature until both fields are answered. Pass checklist: both gaps are named; the action asks a person rather than inventing a value. A weak answer fills in a plausible basis.

Hard. A model decision:

Need:       debug one failed upload for one applicant; the text of that CV may be needed.
Options:    A no access, ask the recruiter to send the file;  B read access to all CVs;
            C access to one named CV, with the recruiter's approval, for 24 hours.
Decision:   C. A is safest but slows a fix; B is far more than the need.
Conditions: access is logged; the person sees no names or contact fields; the CV text is
            not copied outside the support tool; the vendor flow's agreement is on file
            before any access (it is MISSING today, so until then: option A).
Owner:      Head of recruitment approves; privacy owner reviews the log monthly.
Review:     2027-01-15, or after any access request that is refused or abused.

Pass checklist: the need is stated before the grant; "no access" is an option; the decision is narrow, time-limited and logged; it notices the missing vendor agreement; an owner and a review date are named. A weak answer grants broad read access "to be safe".

Summary

• Inventory sources, owners, purposes and permissions; let a program report MISSING and UNCLEAR and never fill a gap.
• Collection authority is not permission for reuse; missing or unclear permission means ask, narrow or substitute, not assume.
• Record provenance and every transformation; a filter changes what later tables can say.
• Removing names is not anonymisation; count unique combinations and test linkage, and treat the result as a screen.
• Check consent quality and power, set retention for copies and logs too, and give each role only the fields its job needs.

Next: Data Rights, Transfers and Records: Requests, Retention and Audit Trails takes up cross-border transfers, vendors and their subcontractors, children's data, copyright questions and a retention schedule you build in Python.