Robust Metrics and Release Gates

Priya in Pune compares two versions of her help-centre assistant. The new one passes 26 of 31 questions, the old one 23. Her lead asks "Can we ship it?" and "it is better" can be wrong in three ways: the gain may be noise, the average may hide a case that must never fail, and a cheaper-looking system may have bought its savings with quality somewhere else.

The previous page designed the evaluation and ended with "version 2 is a candidate, not the winner". This page makes the measurements robust and turns them into a decision: separate measures, intervals and sample-size arithmetic, slices, repeated runs, gaming checks, and a release gate that answers go, review or block with reasons. Everything runs offline on the same 31 synthetic cases with the Beginner fake model, so the numbers describe the method, not a real model.

What you will learn

• How to measure citation support apart from presence, abstention and false-answer rates, and tool use, with regression and adversarial suites kept apart.
• How to report uncertainty on 31 cases: intervals, paired comparisons, the smallest detectable difference, multiple-comparison honesty and repeated runs of a flaky system.
• How to read slices with a small-n guard and catch metric disagreement and gaming.
• How to write a release gate (floors, ceilings, no-regress margins, slice rules, critical cases, a review band), version what a result was measured on, store judge evidence safely and run it all as one CI command.

Run the code from the folder that contains kit/, with export PYTHONPATH=kit; blocks share one namespace. The page builds on Evaluation Design and Judge Calibration (the 31 cases, the plan, the judge), Rubrics, Regression, and Release Gates (its gate has a fixed metric list; ours is general and leaves it untouched) and the bootstrap and sign test of Reranking and Retrieval Evaluation.

One run, many measures

run_report(config) runs the 31 cases through the Beginner RAG pipeline and returns a Report: every rate as a Count (k of n with a Wilson interval, which stays inside 0 to 1 and behaves for small n), a pass flag per case, cost and latency kept apart from quality, and a manifest of versions. "Strict pass" means an answer case needs an answer containing every expected fact and an unanswerable case needs an abstention; the previous page's rule did not read the facts, so its 25 and 27 become 23 and 26.

from evidencedesk.evalkit.suite import BASELINE, CANDIDATE, Config, run_report

old, new = run_report(BASELINE), run_report(CANDIDATE)  # version 1 and version 2 of the previous page
print("dataset", new.manifest.dataset, "| cases", len(new.passed), "| strict passes", sum(old.passed.values()), "->", sum(new.passed.values()))
print(f"{'measure':<20}{'old':>7}{'new':>7}   new rate, 95% interval")
for name in ("answer_correct", "false_abstain_rate", "abstain_correct", "false_answer_rate", "citation_present", "citation_support", "recall_at_3"):
    a, b = old.counts[name], new.counts[name]
    print(f"{name:<20}{f'{a.k}/{a.n}':>7}{f'{b.k}/{b.n}':>7}   {b.rate} {b.interval}")

dataset 8a2bc76d5867 | cases 31 | strict passes 23 -> 26
measure                 old    new   new rate, 95% interval
answer_correct        20/25  22/25   0.88 (0.7, 0.958)
false_abstain_rate     2/25   1/25   0.04 (0.007, 0.195)
abstain_correct         3/6    4/6   0.667 (0.3, 0.903)
false_answer_rate       3/6    2/6   0.333 (0.097, 0.7)
citation_present      23/23  24/24   1.0 (0.862, 1.0)
citation_support      21/21  23/23   1.0 (0.857, 1.0)
recall_at_3           24/25  24/25   0.96 (0.805, 0.993)

Version 2 answers 22 of 25 correctly against 20 and abstains on 4 of 6 against 3; abstention rests on six questions, so its interval runs from 0.30 to 0.90.

Evaluate citation support separately from citation presence

Presence asks "is there a source id that resolves?" The Beginner guard already forces that. Support asks "does the cited text hold the claim?" Here: every expected fact the answer states must appear in a cited chunk. String containment is a proxy; paraphrased claims need a person or a calibrated judge. The fake model copies the top chunk and cites it, so support is trivially perfect. miscite keeps every sentence but cites a neighbouring retrieved chunk: a real id, shown to the model, not holding the claim.

# file: evidencedesk/evalkit/measures.py (excerpt)
def miscite(observed: list[Observed]) -> list[Observed]:
    """A model that writes the right sentence but cites a neighbour: the id is real and was shown, the text does not hold the claim."""
    def swap(o: Observed) -> Observed:
        other = [i for i in o.answer.retrieved_ids if i not in o.cited]
        return replace(o, cited=(other[0],)) if o.answered and other else o
    return [swap(o) for o in observed]

from pathlib import Path

from evidencedesk.evalkit.dataset import load_cases
from evidencedesk.evalkit.measures import measure, miscite, observe
from evidencedesk.evals import CaseModel, build_retriever

cases = load_cases()
obs = observe(cases, build_retriever(Path("kit/corpus"), "sentence", 0.3), CaseModel({}), 0.05)
shifted = miscite(obs)  # same sentences, but each cites the next retrieved chunk
g01, g01_shifted = next(o for o in obs if o.case.id == "g01"), next(o for o in shifted if o.case.id == "g01")
print("g01 cites", g01.cited, "->", g01_shifted.cited)
for label, run in (("as written", obs), ("neighbour cited", shifted)):
    m = measure(run)
    print(f"{label:<16}" + "  ".join(f"{k} {m[k].k}/{m[k].n}" for k in ("citation_present", "citation_support", "answer_correct")))

g01 cites ('kc-returns-v3:c0',) -> ('kc-returns-v3:c1',)
as written      citation_present 24/24  citation_support 23/23  answer_correct 22/25
neighbour cited citation_present 24/24  citation_support 0/23  answer_correct 22/25

Presence stays 24 of 24, support falls to 0 of 23, and answer correctness does not move. A dashboard reporting only "citations valid" calls both systems perfect, which is why the plan lists citation_support as its own metric.

Measure abstention and false-answer rates

Abstention has two errors with different prices. A false answer (answering an unanswerable question) invents a promise; a false abstention (refusing an answerable one) costs a support ticket. They are separate measures over different questions (6 unanswerable, 25 answerable), and the plan rates the first "high". The adversarial set is where false answers hide:

for o in obs:
    if o.case.expect == "abstain" and o.answered:
        print(o.case.id, "| asked:", o.case.question, "| answered from", o.cited[0], "|", o.case.why)
for expect in ("answer", "abstain"):
    rows = [o for o in obs if o.case.expect == expect]
    print(f"{expect:<8} answered {sum(o.answered for o in rows):>2}  abstained {sum(not o.answered for o in rows):>2}")

a02 | asked: What is the warranty on electric motors? | answered from kc-warranty-v2:c0 | Shares words with the warranty page but the topic is absent.
a05 | asked: How do I return a bike to a shop in Brazil? | answered from kc-returns-v3:c0 | Plausible, but the documents do not name the country.
answer   answered 24  abstained  1
abstain  answered  2  abstained  4

Both are the "shares words, lacks the fact" pattern. A better prompt does not fix them; a stricter evidence rule does, at the price of answerable questions, as the grid below shows.

Tool-use evaluation cases

Tool use needs four checks kept apart: selection (right tool, or none), arguments, permission (a tool outside the task's allowlist must not run) and completion (the expected data came back). tool_cases_v1.jsonl holds eight cases, each with what the model "proposed", so no model is called; the calls run through the Beginner registry behind an allowlisted view.

from evidencedesk.evalkit.tooluse import full_registry, load_tool_cases, run_tool_case, tool_measures

tools = load_tool_cases()
print(f"{'case':<5}{'selection':>10}{'args':>6}{'permission':>11}{'complete':>9}  note")
show = lambda v: "-" if v is None else "ok" if v else "FAIL"
for c in tools:
    r = run_tool_case(c, full_registry())
    print(f"{c.id:<5}{show(r['selection']):>10}{show(r['args']):>6}{show(r['permission']):>11}{show(r['complete']):>9}  {c.note[:34]}")
print({k: f"{v.k}/{v.n}" for k, v in tool_measures(tools).items()})

case  selection  args permission complete  note
t01          ok    ok          -       ok  right tool, right arguments
t02          ok  FAIL          -       ok  same sum written in another order:
t03          ok  FAIL          -     FAIL  wrong arguments that the tool itse
t04          ok    ok          -       ok  right tool, right arguments
t05        FAIL     -         ok        -  wrong tool, and it is not allowed 
t06        FAIL     -         ok        -  a greeting needs no tool; calling 
t07          ok     -          -        -  correctly calls nothing
t08          ok    ok          -       ok  an id that does not exist: the too
{'tool_selection': '6/8', 'tool_args': '3/5', 'tool_permission': '2/2', 'tool_complete': '4/5'}

t02 writes the same sum in another order: exact matching marks it wrong although it completes, so normalise arguments before they gate anything. t03 asks for an absurd computation and the tool refuses it. t05 and t06 are wrong or needless calls that the allowlist stops, so permission is 2 of 2. A forbidden call that ran is a security incident; a selection mistake is not.

Regression and adversarial suites

The set field gives two suites with different jobs. The regression suite (golden and production-like, 24 cases) encodes expected behaviour: a flip from pass to fail is a finding whatever the average says. The adversarial suite (7 hostile or tricky inputs) is a pressure gauge: it will not be 100%, so you track its rate and its critical cases and add a case whenever production teaches a new trick. Never average them; a growing adversarial suite would make a healthy system look worse.

from evidencedesk.evalkit.slices import by_suite

for name, report in (("old", old), ("new", new)):
    print(name, {k: f"{v.k}/{v.n}" for k, v in by_suite(report.passed, cases).items()})
print("failing in new, by set:", {s: sorted(c.id for c in cases if c.set == s and not new.passed[c.id]) for s in ("golden", "production_like", "adversarial")})

old {'regression': '19/24', 'adversarial': '4/7'}
new {'regression': '21/24', 'adversarial': '5/7'}
failing in new, by set: {'golden': ['g08'], 'production_like': ['p05', 'p08'], 'adversarial': ['a02', 'a05']}

The adversarial misses (a02, a05) are the false answers above; the regression misses are older.

Statistical caution with small datasets

Thirty-one cases is a smoke test, not a benchmark. Every figure needs its n and its uncertainty. A paired comparison looks at the same questions under both systems, and only questions that changed carry information.

from evidencedesk.evalkit.stats import Count, flips_needed, min_detectable, paired_diff_ci, sample_size
from evidencedesk.retrieval.metrics import paired

ids = sorted(new.passed)
a, b = [float(old.passed[i]) for i in ids], [float(new.passed[i]) for i in ids]
d = paired(dict(zip(ids, a)), dict(zip(ids, b)))
print("old", Count(k=int(sum(a)), n=31).text())
print("new", Count(k=int(sum(b)), n=31).text())
print("better", list(d.better), "worse", list(d.worse), "unchanged", d.tied, "sign-test p", d.p)
print("difference in pass rate", round((sum(b) - sum(a)) / 31, 3), "95% interval", paired_diff_ci(a, b))
print("changed cases needed, all one way, for p < 0.05:", flips_needed())

print("smallest true difference found 8 times in 10, by cases per group:", "  ".join(f"{n}: {min_detectable(n):.2f}" for n in (31, 100, 300, 1000)))
print("cases per group needed to detect 0.10:", sample_size(0.10), "| 0.05:", sample_size(0.05))

old 0.742 (23/31, 95% 0.568-0.863)
new 0.839 (26/31, 95% 0.674-0.929)
better ['a06', 'p02', 'p09'] worse [] unchanged 28 sign-test p 0.25
difference in pass rate 0.097 95% interval (0.0, 0.226)
changed cases needed, all one way, for p < 0.05: 6
smallest true difference found 8 times in 10, by cases per group: 31: 0.28  100: 0.16  300: 0.09  1000: 0.05
cases per group needed to detect 0.10: 251 | 0.05: 1004

Version 2 fixed three questions and broke none: encouraging, but by the sign test p = 0.25, because three flips one way happen a quarter of the time with a fair coin, and the interval for the gain starts at zero. With no regressions it takes six changed questions to reach p below 0.05. The last two lines of the block say how large a difference 31 cases could ever show: two unpaired groups of 31 miss any true difference under about 28 points (a paired test does better when the systems mostly agree, but only if there are flips to count); five points needs about a thousand cases per system. Most differences on a set this size are noise. A gate here can catch large drops, broken critical cases and cheats; it cannot prove small gains.

Multiple comparisons

Try forty-eight settings (two chunkings, six score floors, four margins) on the same 31 cases and pick the best, and you have run forty-eight tests; among twenty, the chance that at least one looks significant at 5% by luck is 64%. Holm's correction compares the smallest p with alpha divided by the number of tests, the next with alpha divided by one fewer, and stops at the first failure. As a control, twenty copies of one deliberately flaky system (40% unparseable replies) are each compared with one reference run of itself.

import itertools

from evidencedesk.evalkit.flaky import FlakyClient
from evidencedesk.evalkit.stats import family_error, holm
from evidencedesk.evals import CaseModel

flat = lambda passed: {i: float(ok) for i, ok in passed.items()}
noisy = lambda seed: run_report(Config("sentence", 0.35, 0.05), client=FlakyClient(CaseModel({}), 0.4, seed)).passed
grid = {}  # 48 settings tried against the same 31 cases
for chunking, score, margin in itertools.product(("paragraph", "sentence"), (0.2, 0.3, 0.35, 0.4, 0.45, 0.5), (0.0, 0.05, 0.1, 0.15)):
    r = run_report(Config(chunking, score, margin))
    grid[f"{chunking[:4]}-{score}-{margin}"] = (sum(r.passed.values()), paired(flat(old.passed), flat(r.passed)))
best = max(grid, key=lambda k: grid[k][0])
d = grid[best][1]
print(f"best of {len(grid)}: {best} passes {grid[best][0]}/31 (old {sum(old.passed.values())}); better {len(d.better)} worse {len(d.worse)} p {d.p}")
print("p < 0.05 before correction:", sum(g[1].p < 0.05 for g in grid.values()), "| after Holm:", sum(holm({k: g[1].p for k, g in grid.items()}).values()))
ref = noisy(1000)  # twenty copies of the SAME system, each compared with one reference run
p = {f"copy{s:02d}": paired(flat(ref), flat(noisy(2000 + s))).p for s in range(20)}
print("identical systems that look different (p < 0.05):", sum(v < 0.05 for v in p.values()), "of 20 | after Holm:", sum(holm(p).values()), "| chance of at least one:", family_error(20))

best of 48: sent-0.35-0.05 passes 27/31 (old 23); better 4 worse 0 p 0.125
p < 0.05 before correction: 0 | after Holm: 0
identical systems that look different (p < 0.05): 1 of 20 | after Holm: 0 | chance of at least one: 0.642

The grid's "winner" passes 27 of 31 against 23, better on four and worse on none, with p = 0.125: not significant even before correction. In the control one of twenty identical systems looks "significantly" different, and Holm removes it. Use a sweep to propose a candidate, then judge that candidate on cases the sweep never saw (the holdout of the previous page, read once).

Flaky and non-deterministic evaluation

Real calls vary even at temperature 0, so one run is one draw. Repeat the run, report the mean and the spread, and list the cases that flip between runs. FlakyClient makes the fake unreliable on purpose (10% unparseable replies, seeded) so the effect is visible and repeatable.

from evidencedesk.evalkit.flaky import repeat_summary

runs = [run_report(Config("sentence", 0.35, 0.05), client=FlakyClient(CaseModel({}), 0.1, seed)).passed for seed in range(10)]
s = repeat_summary(runs)
print({k: s[k] for k in ("runs", "mean", "sd", "min", "max", "stable_pass")}, "always failing:", s["stable_fail"])
print(len(s["flaky"]), "flaky cases, the least stable:", {k: v for k, v in s["flaky"].items() if int(v[0]) <= 7})

{'runs': 10, 'mean': 0.787, 'sd': 0.053, 'min': 0.71, 'max': 0.871, 'stable_pass': 12} always failing: ['a05', 'g08', 'p05', 'p08']
15 flaky cases, the least stable: {'g05': '6/10', 'g07': '7/10'}

The same system passes between 0.71 and 0.87 across ten runs. Version 2's 10-point gain is smaller than that 16-point range, so on a system this flaky a single-run gain means little. The critical case g05 passes only 6 of 10 runs: fix a flaky critical case, do not quarantine it. A flaky non-critical case can be excluded from blocking rules, but keep it in the report with an owner and a date. Noise is not symmetric: an unparseable reply makes an unanswerable case pass by accident.

Provider note (as of 2026-10-12): the IR textbook calls precision at k "the least stable of the commonly used evaluation measures", and NIST's TREC QA page says MRR gives "no credit for multiple correct answers". The book's nDCG uses gain 2^rel - 1 with a log2(1 + rank) discount; other implementations differ, so check the variant before comparing numbers across libraries. Judge-based metrics inherit the biases in the previous page's note; no current judge-human agreement figure for today's models was found on a primary page. Check the current documentation and the papers.

Track slice-level performance

An average hides the group that fails, but every slice is another comparison, so on a small set phantom problems appear. judge_slices shows every slice, judges only those with at least 8 cases, and calls a slice "low" only when the top of its interval is below the overall rate ("weak" when merely 15 points under).

from evidencedesk.evalkit.slices import judge_slices
from evidencedesk.evalkit.stats import Count

overall = Count(k=sum(new.passed.values()), n=len(new.passed))
rows = judge_slices(new.slices, overall)
print("overall", overall.rate, "| judged slices:")
for r in rows:
    if r["status"] != "few":
        print(f"  {r['slice']:<22}{r['k']:>3}/{r['n']:<3} {r['rate']}  {r['interval']}  {r['status']}")
print("not judged (n < 8):", ", ".join(f"{r['slice']} {r['k']}/{r['n']}" for r in rows if r["status"] == "few"))

overall 0.839 | judged slices:
  lang=en                23/28  0.821  (0.644, 0.921)  ok
  task=lookup            12/12  1.0  (0.757, 1.0)  ok
  set=golden             13/14  0.929  (0.685, 0.987)  ok
  set=production_like     8/10  0.8  (0.49, 0.943)  ok
not judged (n < 8): lang=bn 3/3, task=exact_term 3/3, task=multi_fact 5/6, task=paraphrase 2/4, task=unanswerable 4/6, set=adversarial 5/7

Only four slices can be judged. Bangla (3 cases), exact terms (3), paraphrase (4) and unanswerable (6) are exactly where the risk is, and the set is blind there. That is a finding: collect more cases in those slices, do not loosen the guard.

Metric disagreement and gaming

When one metric improves and another worsens, that is a trade-off for a person to decide, not an error to hide. A single optimised number also invites cheating by the system or the team (Goodhart's law: a measure that becomes a target stops being a good measure). Pair each gameable metric with a counter-metric (abstention with answer correctness, citation presence with support) and test the gate with systems that cheat on purpose.

from evidencedesk.evalkit.goodhart import disagreements, gaming_flags

harsh = run_report(Config("sentence", 0.5, 0.05))
print("version 2 -> min_score 0.5:", [d for d in disagreements(new.counts, harsh.counts) if "_rate" not in d])
trivial = {"always abstains": run_report(Config("sentence", 9.0, 0.0)), "never abstains": run_report(Config("sentence", 0.0, -1.0))}
for name, r in trivial.items():
    c = r.counts
    print(f"{name:<16} strict passes {sum(r.passed.values()):>2}/31  answer_correct {c['answer_correct'].rate}  abstain_correct {c['abstain_correct'].rate}", gaming_flags(new.counts, c))

version 2 -> min_score 0.5: ['abstain_correct +0.33 but answer_correct -0.20']
always abstains  strict passes  6/31  answer_correct 0.0  abstain_correct 1.0 ['abstain_correct rose +0.33 while answer_correct fell -0.88']
never abstains   strict passes 22/31  answer_correct 0.88  abstain_correct 0.0 []

Raising the score floor to 0.5 improves abstention by 33 points and costs 20 points of answer correctness: disagreement, flagged. The always-abstain system has a perfect abstention rate and gaming_flags catches it. The never-abstain system matches version 2's answer correctness exactly (0.88), so that metric alone cannot tell a careful system from one that never refuses. A gate that either trivial system passes is broken.

Quality gates for release

A gate turns measurements into go, review or block. A gate on averages fails twice: a critical case can fail while the mean rises, and a floor on a small sample is unprovable. The policy is data (release_policy_v1.json) with these rules:

• Critical cases (four ids): any failure, or absence, blocks whatever the averages say.
• Floors and ceilings per measure with an action, for quality counts and for cost or latency alike; a missing measure is never silently passed.
• Review band: a value within band above a floor passes on the point estimate but goes to a person. Floor plus band equals the plan's threshold (a test checks this against the previous page's plan).
• No-regress margins against the baseline, and a review listing every case that passed in the baseline and fails now.
• Slice rule for judged slices; unjudged ones are noted. A note never changes the verdict: a floor met on the point estimate whose interval starts below it is marked "unproven".

Three candidates against the stored baseline (version 1): version 2, the grid's winner (score floor 0.35) and a harsh setting (0.5).

from evidencedesk.evalkit.gates import evaluate, load_policy

policy = load_policy()
candidates = {"old (v1)": BASELINE, "new (v2)": CANDIDATE, "score 0.35": Config("sentence", 0.35, 0.05), "score 0.5": Config("sentence", 0.5, 0.05)}
reports = {name: run_report(cfg) for name, cfg in candidates.items()}
for name, r in list(reports.items())[1:]:  # the baseline is not judged against itself
    d = evaluate(policy, r, old)
    print(f"{name:<11}{d.verdict.upper()}")
    for x in d.reasons:
        if x.level != "note":
            print(f"   {x.level:<7}{x.rule}: {x.text}")
print("trivial systems:", {n: evaluate(policy, r, old).verdict for n, r in trivial.items()})

new (v2)   BLOCK
   block  critical: case a02 failed
   block  floor: abstain_correct 0.667 (4/6) is below the floor 0.8
score 0.35 REVIEW
   review band: abstain_correct 0.833 (5/6) is inside the review band above the floor 0.8
score 0.5  BLOCK
   block  critical: case g05 failed
   block  floor: answer_correct 0.68 (17/25) is below the floor 0.8
   block  regress: answer_correct fell from 0.8 to 0.68 (allowed drop 0.05)
   review flips: 3 cases passed in the baseline and fail now: a01, g07, p06
trivial systems: {'always abstains': 'block', 'never abstains': 'block'}

Version 2 improves the average and is still blocked, by the abstention floor and by a02, a critical case. The 0.35 setting is not blocked: abstention at 5 of 6 sits in the review band, so a person decides with the notes attached. The harsh setting has the best abstention (6 of 6) and is blocked by a critical retrieval case, an answer floor and a no-regress rule at once. Both trivial systems are blocked. The band is the honest middle: "passes on the numbers, and the numbers are thin".

What the gate can and cannot prove

It cannot prove a floor with 25 cases. The same rate with ten times the cases (illustrative arithmetic, not a measurement) shows what proof needs:

from evidencedesk.evalkit.stats import Count

for times in (1, 10):
    c = Count(k=22 * times, n=25 * times)
    print(f"answer_correct 0.88 from {c.n} cases: interval {c.interval} -> floor 0.8 proven: {c.interval[0] >= 0.8}")

answer_correct 0.88 from 25 cases: interval (0.7, 0.958) -> floor 0.8 proven: False
answer_correct 0.88 from 250 cases: interval (0.834, 0.915) -> floor 0.8 proven: True

That is why unproven floors become notes and bands rather than silent passes, and why every verdict states n.

Version prompts, models and retrieval settings

The Manifest in every report holds the dataset hash, the tool-case set, the metrics version, the answer-prompt hash, the model name, the retrieval settings and, when a judge produced a number, the judge prompt hash; its sha names the configuration.

from evidencedesk.evalkit.versions import changed, not_comparable

mid = reports["score 0.35"]
m = mid.manifest
print("manifest", m.sha, "| dataset", m.dataset, "| prompt", m.prompt_sha, "| model", m.model, "| retrieval", m.retrieval)
print("changed against the baseline:", changed(old.manifest, m))
print("numbers comparable:", not_comparable(old.manifest, m) == [], "| after editing one case:", not_comparable(old.manifest, m.model_copy(update={"dataset": "0123456789ab"})))

manifest 5cb05a77bd | dataset 8a2bc76d5867 | prompt c3bcffe3e0 | model fake-case-model | retrieval {'chunking': 'sentence', 'min_score': 0.35, 'min_margin': 0.05, 'k': 3}
changed against the baseline: {'retrieval.chunking': ('paragraph', 'sentence'), 'retrieval.min_score': (0.2, 0.35)}
numbers comparable: True | after editing one case: ['dataset differs (8a2bc76d5867 vs 0123456789ab)']

Two kinds of difference exist. Prompt, model and retrieval settings are what you test (changed lists them). The dataset, tool cases and metric definitions are what you measure with: if they differ, not_comparable says so and the gate sends the run to review. A baseline is a saved report plus a reason and the manifest hash it replaced, written by an explicit command and never as a side effect: --init records what exists and approves nothing, and --promote REASON refuses when the gate says block.

Store judge evidence and disagreements

A verdict that cannot be reviewed later cannot be trusted. Per pair the evidence record keeps the judge prompt hash, the verdict in both orders and after the order check, the human label, the three dimension labels, a structured reason made of scores and a hash of the judged text. It keeps no answer text. A reviewer note is allowed, scrubbed (secrets, email addresses, long digit runs) and cut to 160 characters.

from evidencedesk.evalkit.evidence import collect, review_queue, scrub, with_note
from evidencedesk.evalkit.judge import DEMO_JUDGES, load_pairs, load_prompts

pairs, prompts = load_pairs(), load_prompts()
ev = collect(DEMO_JUDGES["mixed"], prompts[1], pairs)
e = ev[4]
print(e.pair_id, e.lang, "| order a-b:", e.order_ab, "| order b-a:", e.order_ba, "| judged:", e.judged, "| human:", e.human, "|", e.why, "| text hash", e.input_sha)
queue = review_queue(ev)
print(len(queue), "of", len(ev), "pairs go to a person:", {r: sum(r in why for why in queue.values()) for r in ("order flip", "judge vs human")})
note = "call " + "+44 7911 " + "123456 or " + "ana" + "@" + "example.org " + "token" + "=" + "fake-value-123"  # bait made at run time, never stored
print(with_note(ev[4], note).note)

clear-5 bn | order a-b: tie | order b-a: b | judged: tie | human: a | score a 0.15 b 0.1 | text hash 84e5cd8fc5a1
17 of 30 pairs go to a person: {'order flip': 17, 'judge vs human': 12}
call +[number] or [email] [secret]

Seventeen of thirty pairs go to a person: every order flip, and twelve against the human label. The record is a Bangla pair where the mixed judge says "tie" in one order and "b" in the other while the humans said "a". The bait was assembled at run time, so no personal-looking string sits in the kit. Redaction is a safety net: log structured fields, not free text, and set a retention period.

Cost and latency beside quality

Cost and latency are not quality metrics, and a cheaper system is better only if the same evaluation says its quality held. They are separate scalars with their own ceilings. Tokens are estimated from request text; latency is a formula (15 ms retrieval plus 400 ms and 2 ms per input token per model call), because no clock is trusted offline.

print(f"{'candidate':<12}{'tokens/case':>12}{'calls/case':>11}{'p95 ms (formula)':>17}{'answer_correct':>16}  gate")
for name, r in reports.items():
    s = r.scalars
    print(f"{name:<12}{s['input_tokens_per_case']:>12}{s['model_calls_per_case']:>11}{s['latency_p95_ms']:>17}{r.counts['answer_correct'].rate:>16}  {evaluate(policy, r, old).verdict}")

candidate    tokens/case calls/case p95 ms (formula)  answer_correct  gate
old (v1)           241.9      0.839           1065.0             0.8  block
new (v2)           223.2      0.839           1005.0            0.88  block
score 0.35         205.5      0.806           1003.0            0.88  review
score 0.5          128.4      0.581            987.0            0.68  block

The harsh setting uses 42% fewer tokens than version 2 and nearly a third fewer calls because it refuses more, and loses 20 points of answer correctness. Cost alone would have called it the winner.

Mini-lab: evaluation and release gate

python -m evidencedesk.evalkit.ci runs the fixed cases, applies the policy to the baseline and prints versions, a metric table with intervals and the decision. Exit code 1 means block; go and review exit 0, because a review asks a person and is not a failed build. Flags set the candidate (--chunking, --min-score, --min-margin).

python -m evidencedesk.evalkit.ci | grep -vE "  note|tool_|input_"; echo "exit ${PIPESTATUS[0]}"

policy evidencedesk-release v1  candidate 0f9236807b  baseline c4a8abc04e
changed: retrieval.chunking paragraph->sentence, retrieval.min_score 0.2->0.3
metric                     value  95% interval   baseline
answer_correct              0.88  0.7-0.958           0.8
abstain_correct            0.667  0.3-0.903           0.5
citation_support             1.0  0.857-1.0           1.0
recall_at_3                 0.96  0.805-0.993        0.96
latency_p95_ms            1005.0                   1065.0
GATE: BLOCK
  block  critical: case a02 failed
  block  floor: abstain_correct 0.667 (4/6) is below the floor 0.8
exit 1

Version 2 is blocked, exit 1. Now the grid's setting and the harsh one (verdict lines only):

python -m evidencedesk.evalkit.ci --min-score 0.35 | grep -E "^GATE|  (block|review)"; echo "exit ${PIPESTATUS[0]}"
python -m evidencedesk.evalkit.ci --min-score 0.5 | grep -E "^GATE|  (block|review)"; echo "exit ${PIPESTATUS[0]}"

GATE: REVIEW
  review band: abstain_correct 0.833 (5/6) is inside the review band above the floor 0.8
exit 0
GATE: BLOCK
  block  critical: case g05 failed
  block  floor: answer_correct 0.68 (17/25) is below the floor 0.8
  block  regress: answer_correct fell from 0.8 to 0.68 (allowed drop 0.05)
  review flips: 3 cases passed in the baseline and fail now: a01, g07, p06
exit 1

The tests that pin this page (explicit names, so the count does not drift when other pages add tests):

python -m pytest kit/tests/test_robust_stats.py kit/tests/test_robust_measures.py kit/tests/test_robust_gates.py | sed 's/ in [0-9.]*s//'

..........................                                               [100%]
26 passed

New in the kit: eleven modules in evidencedesk/evalkit/ (stats, measures, tooluse, slices, flaky, goodhart, versions, gates, evidence, suite, ci), three data files and three test_robust_*.py files; the shipped baseline_v1.json is checked by a test against the code that made it. The Beginner gate.py is unchanged. The evaluation chapter is complete for now.

Where this shows up at work

Mateo in Bogotá runs a claims assistant. Every pull request showed "pass rate 94%"; after a prompt change it read 95% while a refund-limit case quietly began returning the wrong ceiling. Now the pipeline prints counts with intervals, blocks on a short list of critical cases, reviews every pass-to-fail flip and keeps the baseline and its reason in the repository.

Common mistakes

• Reporting a rate without n. "Abstention 83%" is 5 of 6. Carry counts, show intervals, refuse to judge small slices.
• Treating a sweep winner as a result. Correct for the number of comparisons and confirm on unseen cases.
• Gating on the average alone. A mean can rise while a must-not-fail case starts failing. Name critical cases and test the gate with cheating systems.
• Comparing across changed yardsticks. A new dataset hash or metric definition makes numbers incomparable; store the manifest and refuse.
• Logging the judged text. Keep hashes, scores and scrubbed short notes; set retention.

Try it yourself

1. Easy (revisits Evaluation Datasets and Deterministic Tests): a candidate fixes 5 of the 31 cases and breaks none. Significant at 5%? And 6?
2. Medium (revisits Rubrics, Regression, and Release Gates): a language slice passes 60% of its cases; the overall rate is 26/31. From which slice size is it "low", not just "weak"?
3. Hard (revisits Evaluation Design and Judge Calibration): write with_judge(decision, record, prompt, pairs): a stale calibration sends the run to review, a kappa under 0.6 blocks. Try the mixed judge with prompt 1 and the clean judge with prompts 1 and 2.

Answers

Easy. Five one-way flips give p = 0.062, six give 0.031.

a = {f"c{i:02d}": 0.0 for i in range(31)}
for fixed in (5, 6):
    b = {k: 1.0 if int(k[1:]) < fixed else 0.0 for k in a}
    print(f"{fixed} cases fixed, none broken: p = {paired(a, b).p}")

5 cases fixed, none broken: p = 0.062
6 cases fixed, none broken: p = 0.031

Pass checklist: only changed cases counted; two-sided exact test; you said significance still assumes the cases are a fair sample.

Medium.

from evidencedesk.evalkit.slices import judge_slices
from evidencedesk.evalkit.stats import Count

for n in range(8, 20):
    row = judge_slices({"lang=bn": Count(k=round(0.6 * n), n=n)}, Count(k=26, n=31))[0]
    print(n, f"{row['k']}/{n}", row["status"])
    if row["status"] == "low":
        break

8 5/8 weak
9 5/9 low

At 8 cases the interval still reaches the overall rate ("weak"); at 9 its top falls below it ("low"). Pass checklist: minimum size and interval rule are separate checks; you noticed the result moves with the overall rate.

Hard.

from evidencedesk.evalkit.calibrate import calibrate, is_stale
from evidencedesk.evalkit.gates import Decision, Reason


def with_judge(decision, record, prompt, pairs, min_kappa=0.6):
    if is_stale(record, prompt, pairs):
        extra = Reason("review", "judge", "calibration is stale")
    elif record["kappa"] < min_kappa:
        extra = Reason("block", "judge", f"kappa {record['kappa']} is below {min_kappa}")
    else:
        return decision
    reasons = decision.reasons + (extra,)
    return Decision("block" if any(r.level == "block" for r in reasons) else "review", reasons)


mid = evaluate(policy, reports["score 0.35"], old)
for judge, version in (("mixed", 1), ("clean", 1), ("clean", 2)):
    record = calibrate(pairs, DEMO_JUDGES[judge], prompts[1], swap=False)  # measured with prompt 1
    d = with_judge(mid, record, prompts[version], pairs)
    print(f"{judge} judge, prompt v{version}:", d.verdict, [r.text for r in d.reasons if r.rule == "judge"])

mixed judge, prompt v1: block ['kappa 0.456 is below 0.6']
clean judge, prompt v1: review []
clean judge, prompt v2: review ['calibration is stale']

The mixed judge fails on kappa and blocks. The clean judge passes on the prompt it was calibrated with and goes to review when the prompt changes, because the old calibration describes a different judge. Pass checklist: staleness is checked before kappa; the judge is re-calibrated with the new prompt before it returns to a gate.

Summary

• Measure citation support apart from presence, false answers apart from false abstentions, and the four tool checks apart; keep regression and adversarial suites separate.
• On 31 cases carry counts and intervals, compare paired, know how large the smallest detectable difference is, correct for sweeps and repeat flaky runs.
• Judge slices only above a minimum size; pair gameable metrics with counter-metrics and block trivial cheaters.
• A gate has critical cases, floors, ceilings, a review band, no-regress margins and slice rules; the verdict states reasons, and an unproven floor is a note, not a pass.
• Version dataset, prompt, model and retrieval settings in a manifest, promote baselines deliberately, keep judge evidence without the text, and put cost and latency beside, never inside, quality.

Next: Fine-Tuning Decisions and LoRA Concepts. The evaluation chapter is complete for now: you can measure, compare and gate. The next page uses that to decide whether fine-tuning is the right fix at all, and what a baseline must contain before any training.