ক্যাপস্টোন, চিট শিট ও এরপর কী: গো/নো-গো মূল্যায়ন প্যাক

School:   Brightside Language School (invented), evening English classes for adults.
Feature:  HomeworkNote drafts a short feedback note on each written homework and flags
          homework that may need a teacher's attention. A teacher edits and sends every note.
People:   Head of Studies, six teachers, about 100 learners, reception staff.
Question: should the school pilot it for one term, and on what conditions?

# file: pack/README.md
# Review pack: HomeworkNote (invented)
Owner: Head of Studies (a role)
Review date: 2027-03-01

## Purpose
Draft feedback notes on adult learners' homework and flag homework that may need a teacher.
A teacher edits and sends every note.

## Scope
One school, evening English classes, written homework, about 100 learners.

## Out of scope
Marking, ranking learners, any decision about fees or places.

## Users
Six teachers, trained on the tool.

## Affected non-users
Learners (they see the note, not the flag) and reception staff.

## Benefits
Teachers save reading time; notes arrive within a day.

## Harms
A missed flag leaves a learner without help; a wrong note can discourage.

## Assumptions
The teacher's later judgement of "needed help" is a fair label (untested).

## Alternatives
Teachers write all notes by hand, or use a shared list of common comments.

## Recommendation
proceed with conditions

## Rationale
Time is saved, but the toy check shows a group gap and the evidence is thin.

## Conditions
One-term pilot; a teacher reads every homework; a real group check at the end; the stop rule.

# file: pack/risk_register.json
[
 {"id": "R1", "text": "Learners who need help are missed, more often in one group",
  "likelihood": 3, "severity": 4, "owner": "Head of Studies",
  "controls": ["teacher reads every homework", "weekly group check"],
  "res_likelihood": 2, "res_severity": 3, "evidence": "toy data only"},
 {"id": "R2", "text": "Homework text goes to the vendor without a checked permission",
  "likelihood": 2, "severity": 4, "owner": "Privacy adviser",
  "controls": ["no names in the text sent", "vendor asked about retention"],
  "res_likelihood": 2, "res_severity": 3, "evidence": "opinion"}
]

# file: pack/toy_data.csv
group,label,flag,count
english,1,1,18
english,1,0,4
english,0,1,5
english,0,0,33
other,1,1,9
other,1,0,7
other,0,1,4
other,0,0,20
unrecorded,1,1,2
unrecorded,1,0,1
unrecorded,0,1,0
unrecorded,0,0,3

# file: pack/fairness.py
import csv
from pathlib import Path

MIN_EACH = 10   # a chosen limit: with fewer positives or negatives, no rate is reported
rows = list(csv.DictReader(open(Path(__file__).resolve().parent / "toy_data.csv")))
print("Toy data: a calculation only, not any real group.")
found = {}
for g in sorted({r["group"] for r in rows}):
    c = {(int(r["label"]), int(r["flag"])): int(r["count"]) for r in rows if r["group"] == g}
    tp, fn, fp, tn = c[1, 1], c[1, 0], c[0, 1], c[0, 0]
    n, pos, neg = tp + fn + fp + tn, tp + fn, fp + tn
    if min(pos, neg) < MIN_EACH:
        print(f"{g:<11}n={n:<4}NOT REPORTED ({pos} needed help, {neg} did not; minimum {MIN_EACH})")
    else:
        found[g] = tp / pos
        print(f"{g:<11}n={n:<4}flag rate {(tp + fp) / n:.2f}   flagged among those who needed help {tp / pos:.2f}")
print(f"gap in that last rate, english minus other: {found['english'] - found['other']:.2f}")

# file: pack/system_card.json
{"name": "HomeworkNote (invented)", "version": "0.1", "as_of": "2026-10-12",
 "known": ["drafts a note from homework text", "flags homework", "a teacher sends every note"],
 "unknown": [{"what": "accuracy on handwriting scans", "why": "never tested"},
             {"what": "accuracy by group on real data", "why": "only toy counts so far"}],
 "data_limits": "toy counts only; the unrecorded group has 3 who needed help, so no rate"}

# file: pack/test_plan.json
[
 {"control": "C1", "claim": "a teacher reads every homework before a note is sent",
  "evidence": "weekly sample of 20 notes", "criterion": "at least 19 show a teacher edit",
  "owner": "Head of Studies", "result": "NOT TESTED"},
 {"control": "C2", "claim": "the flag misses learners in one group no more than in another",
  "evidence": "fairness.py on the term's data", "criterion": "gap in the last rate at most 0.10",
  "owner": "Data lead", "result": "FAIL"}
]

# file: pack/oversight.md
## Oversight
A teacher reads the homework and the draft and may change or delete the note. The flag never
changes a mark or a place.

## Appeal
A learner may ask the Head of Studies for a new note from another teacher, answered in 14 days.

## Stop-use
Stop, and write notes by hand, if the C2 gap is above 0.20 twice in a row, a note is sent
unread, or a privacy incident is open. Owner: Head of Studies.

# file: pack/incident_plan.md
## Severity
S1 rights harm or data exposure; S2 many learners or cause unknown; S3 small and reversible;
S4 inconvenience. Unknown facts raise the level.

## Containment
Flag off, model version pinned, notes written by hand. Owner: incident lead.

## Evidence
Versions, times, counts, redacted samples. No homework text in chat.

## Post-incident review
Blameless, within 10 working days; each action has one owner, a date and a check.

# file: pack/frameworks.json
[
 {"framework": "NIST AI RMF", "version": "1.0 (2023, voluntary)", "review_date": "2026-12-01",
  "maps": {"Govern": "oversight.md", "Map": "risk_register.json", "Measure": "test_plan.json",
           "Manage": "incident_plan.md"}},
 {"framework": "School AI-use policy", "version": "2026-03 (company policy)",
  "review_date": "2027-03-01", "maps": {"Govern": "README.md"}}
]

# file: pack/open_questions.md
- Q1 | status: OPEN | asked: the school's privacy adviser | May homework text go to the vendor?
- Q2 | status: OPEN | asked: a lawyer in our country | Do local rules cover automated notes?

# file: packcheck.py
import json, subprocess, sys
from datetime import date
from pathlib import Path

AS_OF = date(2026, 10, 12)   # a fixed "today", so every run repeats
ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else "pack")

def text(name):
    p = ROOT / name
    return p.read_text(encoding="utf-8") if p.is_file() else ""

def data(name):
    try:
        return json.loads(text(name))
    except ValueError:
        return None

def sections(name):
    out, cur = {}, None
    for line in text(name).splitlines():
        if line.startswith("## "):
            cur = line[3:].strip()
            out[cur] = ""
        elif cur:
            out[cur] += line + "\n"
    return {k: v.strip() for k, v in out.items()}

def verdict(missing):
    if missing:
        return "MISSING", "empty or absent: " + ", ".join(missing)
    return "PASS", "found, filled"

def sec(name, wanted):
    s = sections(name)
    return verdict([w for w in wanted if not s.get(w)])

def rows(name, fields):
    d = data(name)
    if not isinstance(d, list) or not d:
        return d, ("MISSING", name + " absent or empty")
    gaps = [f"{r.get('id', r.get('control'))}.{f}" for r in d for f in fields if not r.get(f)]
    return d, verdict(gaps)

def fairness():
    try:
        p = subprocess.run([sys.executable, "-I", "fairness.py"], cwd=ROOT,
                           capture_output=True, text=True, timeout=20)
        return p.returncode == 0 and bool(p.stdout.strip()), p.stdout
    except Exception:
        return False, ""

def c05():
    if len(text("toy_data.csv").splitlines()) < 2:
        return "MISSING", "toy_data.csv absent or empty"
    return verdict([] if (data("system_card.json") or {}).get("data_limits") else ["data_limits"])

def c06():
    card = data("system_card.json") or {}
    gaps = [k for k in ("known", "unknown") if not card.get(k)]
    gaps += [f"unknown[{i}].why" for i, u in enumerate(card.get("unknown", [])) if not u.get("why")]
    return verdict(gaps)

def c10():
    d, v = rows("frameworks.json", ["framework", "version", "review_date"])
    for r in d if v[0] == "PASS" else []:
        try:
            late = date.fromisoformat(r["review_date"]) < AS_OF
        except ValueError:
            return "NEEDS-HUMAN-REVIEW", r["framework"] + ": review_date is not a date"
        if late or "UNKNOWN" in r["version"]:
            return "NEEDS-HUMAN-REVIEW", r["framework"] + (": review date passed" if late else ": version unknown")
    return v

def c11():
    lines = [x for x in text("open_questions.md").splitlines() if x.startswith("- ")]
    bad = [x.split("|")[0].strip("- ") for x in lines if "status:" not in x or "asked:" not in x]
    return verdict(["status or asked on " + ", ".join(bad)] if bad else [] if lines else ["questions"])

def c12():
    s = sections("README.md")
    rec = s.get("Recommendation", "").lower()
    if rec not in ("proceed", "proceed with conditions", "do not proceed"):
        return "MISSING", "Recommendation is not one of the three phrases"
    if not s.get("Rationale") or (rec.endswith("conditions") and not s.get("Conditions")):
        return "MISSING", "Rationale or Conditions empty"
    return "NEEDS-HUMAN-REVIEW", "present; a person judges the reasoning"

def c13():
    a, b = fairness(), fairness()
    if not (a[0] and b[0]):
        return "MISSING", "the offline script did not run"
    return ("PASS", "two runs, same output") if a == b else ("NEEDS-HUMAN-REVIEW", "outputs differ")

CHECKS = [
    ("scope", lambda: sec("README.md", ["Purpose", "Scope", "Out of scope", "Users", "Affected non-users"])),
    ("benefits", lambda: sec("README.md", ["Benefits", "Harms", "Assumptions", "Alternatives"])),
    ("risks", lambda: rows("risk_register.json", ["owner", "controls", "res_likelihood", "res_severity"])[1]),
    ("fairness", lambda: ("NEEDS-HUMAN-REVIEW", "it runs; a person checks the figures")
        if fairness()[0] else ("MISSING", "fairness.py absent or failed")),
    ("data limits", c05), ("known/unknown", c06),
    ("test plan", lambda: rows("test_plan.json", ["claim", "evidence", "criterion", "owner", "result"])[1]),
    ("oversight", lambda: sec("oversight.md", ["Oversight", "Appeal", "Stop-use"])),
    ("incidents", lambda: sec("incident_plan.md", ["Severity", "Containment", "Evidence", "Post-incident review"])),
    ("frameworks", c10), ("legal", c11), ("recommendation", c12), ("reproducible", c13)]
tally = {"PASS": 0, "NEEDS-HUMAN-REVIEW": 0, "MISSING": 0}
print("packcheck:", ROOT.name, "as of", AS_OF)
for n, (title, check) in enumerate(CHECKS, 1):
    status, note = check()
    tally[status] += 1
    if status != "PASS" or "--problems" not in sys.argv:
        print(f"{n:02d} {status:<19}{title}: {note}")
print(tally)
print("PASS means a file was found with its fields filled. It is not a quality verdict.")

import subprocess, sys

def run(*args):
    out = subprocess.run([sys.executable, "packcheck.py", *args], capture_output=True, text=True)
    print(out.stdout, end="")

run("pack")

packcheck: pack as of 2026-10-12
01 PASS               scope: found, filled
02 PASS               benefits: found, filled
03 PASS               risks: found, filled
04 NEEDS-HUMAN-REVIEW fairness: it runs; a person checks the figures
05 PASS               data limits: found, filled
06 PASS               known/unknown: found, filled
07 PASS               test plan: found, filled
08 PASS               oversight: found, filled
09 PASS               incidents: found, filled
10 PASS               frameworks: found, filled
11 PASS               legal: found, filled
12 NEEDS-HUMAN-REVIEW recommendation: present; a person judges the reasoning
13 PASS               reproducible: two runs, same output
{'PASS': 11, 'NEEDS-HUMAN-REVIEW': 2, 'MISSING': 0}
PASS means a file was found with its fields filled. It is not a quality verdict.

import shutil
from pathlib import Path

shutil.copytree("pack", "pack_damaged")
d = Path("pack_damaged")

def edit(name, old, new):
    p = d / name
    text = p.read_text()
    assert old in text, name
    p.write_text(text.replace(old, new))

(d / "toy_data.csv").unlink()
edit("oversight.md", "## Appeal", "## Notes")
edit("test_plan.json", "gap in the last rate at most 0.10", "")
edit("frameworks.json", "1.0 (2023, voluntary)", "UNKNOWN")
edit("open_questions.md", "| status: OPEN | asked: a lawyer", "| asked: a lawyer")
run("pack_damaged", "--problems")

packcheck: pack_damaged as of 2026-10-12
04 MISSING            fairness: fairness.py absent or failed
05 MISSING            data limits: toy_data.csv absent or empty
07 MISSING            test plan: empty or absent: C2.criterion
08 MISSING            oversight: empty or absent: Appeal
10 NEEDS-HUMAN-REVIEW frameworks: NIST AI RMF: version unknown
11 MISSING            legal: empty or absent: status or asked on Q2
12 NEEDS-HUMAN-REVIEW recommendation: present; a person judges the reasoning
13 MISSING            reproducible: the offline script did not run
{'PASS': 5, 'NEEDS-HUMAN-REVIEW': 2, 'MISSING': 6}
PASS means a file was found with its fields filled. It is not a quality verdict.