Chapter 7 · Basic Evals and Regression Testing
Evaluation Datasets and Deterministic Tests
- Page 13 of 22
- 24 min read
Last week Nadia in Dhaka changed one line in a prompt. The example she tried still worked, so she shipped it. Two days later support found that three other questions had quietly broken. A feature that works on one example can fail on the next; evaluation turns "it seems fine" into numbers you can compare before and after a change. This page builds the foundation: stored cases, a dataset you can fingerprint, tests that need no model, and measurements of retrieval, citations and abstention. The code is one new module, evidencedesk/evals.py, built on earlier pages. The model is a fake again: hand-written replies make runs repeatable. Real models are less predictable, which is exactly why you test.
What you will learn
- How to store an evaluation case with its reason, and keep cases in a versioned dataset with a drift check.
- Which checks are exact and which need judgment, and how to test parsing, retrieval, citations, tools and fallbacks without a model.
- How to measure retrieval recall, citation validity and abstention, and why quality dimensions are scored separately.
- How to save a baseline before a change and compare two strategies on the same dataset.
Define an evaluation case
An evaluation case has an input, an expected behaviour and a reason: the sentence that tells a future reader why the case exists and what its failure means. Without it nobody dares delete a case. RAG Reliability and Evaluation Data gave us RagCase. EvalCase wraps it and adds two kinds: tool cases (a tool call and the expected result or error code) and format cases (a raw model reply and the expected parse result).
# file: kit/evidencedesk/evals.py (new file)
"""Evaluation: cases, a versioned dataset with a drift check, deterministic runners and metrics, a saved baseline. No live model."""
from __future__ import annotations
import hashlib
import json
import re
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, ValidationError, model_validator
from evidencedesk.chunking import chunk_corpus
from evidencedesk.citations import MIN_MARGIN, RagCase, check_citations, guarded_answer, load_rag_cases
from evidencedesk.llm import LLMRequest, LLMResponse, Usage
from evidencedesk.rag import MIN_SCORE, RecordedReplies, Retrieved, Retriever, reply
from evidencedesk.results import parse_answer
from evidencedesk.tools import ToolCall, ToolRegistry
KIT = Path(__file__).resolve().parent.parent
REASONS = { # why a RagCase matters, by its first tag (the RAG file has no reason column)
"direct": "The plain question customers ask most: a miss here is a basic failure.",
"exact_term": "An exact identifier or number: hard for embeddings.",
"paraphrase": "Different words, same meaning: retrieval beyond shared vocabulary.",
"bangla": "A Bangla question must find a Bangla document.",
"stale": "A superseded policy says something else: the old version must never answer.",
"unanswerable": "The corpus has no answer: the assistant must say so, not guess.",
}
# --- 1. One evaluation case: input, expected behaviour, and why it matters ---
class EvalCase(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
id: str
kind: Literal["rag", "tool", "format"]
input: str | dict[str, Any] # rag: the question. tool: {name, arguments}. format: {text, stop_reason}
expect: str # rag: answer | abstain. tool: ok or an error code. format: ok or a parse code
doc_ids: list[str] = [] # rag: documents that hold the answer
must_cite: bool = False
tags: list[str] = []
reason: str = Field(min_length=10) # why this case exists
@model_validator(mode="after")
def consistent(self) -> EvalCase:
if self.kind == "rag":
if self.expect not in ("answer", "abstain") or not isinstance(self.input, str):
raise ValueError("a rag case needs a question and expect answer or abstain")
if (self.expect == "answer") != bool(self.doc_ids) or (self.must_cite and self.expect != "answer"):
raise ValueError("answer needs doc_ids (and only then), and only an answer can require a citation")
elif not isinstance(self.input, dict):
raise ValueError(f"a {self.kind} case needs an object as input")
return self
@classmethod
def from_rag(cls, case: RagCase, reason: str | None = None) -> EvalCase:
return cls(id=case.id, kind="rag", input=case.question, expect="answer" if case.answerable else "abstain",
doc_ids=case.expected_doc_ids, must_cite=case.must_cite, tags=list(case.tags),
reason=reason or REASONS[case.tags[0]])The validator rejects a bad case: a RAG case that expects an answer must name its documents, and every case needs a reason of at least ten characters. from_rag converts the 16 existing cases without copying them; the reason comes from the first tag.
from pathlib import Path
from pydantic import ValidationError
from evidencedesk.citations import load_rag_cases
from evidencedesk.evals import EvalCase
stale = EvalCase.from_rag(load_rag_cases(Path("kit/data/rag_cases.jsonl"))[8]) # rc09, the stale trap
print(stale.model_dump_json(indent=1))
try:
EvalCase(id="x1", kind="rag", input="Can I pay in instalments?", expect="abstain", reason="because")
except ValidationError as error:
print("rejected:", error.errors()[0]["msg"]){
"id": "rc09",
"kind": "rag",
"input": "Are refunds paid as store credit?",
"expect": "answer",
"doc_ids": [
"kc-returns-v3"
],
"must_cite": true,
"tags": [
"stale"
],
"reason": "A superseded policy says something else: the old version must never answer."
}
rejected: String should have at least 10 charactersSeparate exact tests from judgment tests
Mixing checkable and unchecked questions gives tests that fail at random or prove nothing. The rule: if you can write the expected value before running the code, the test is exact; if you would have to read the output and decide, it is a judgment test.
| Question about the output | Kind | Example |
|---|---|---|
| Is it valid JSON, with the right fields and a finished stop reason? | Exact | format cases (invalid_json, truncated) |
| Does the tool refuse extra or malformed arguments? | Exact | tool cases (invalid_arguments) |
| Is the expected document in the top 3? | Exact (the labels are human work) | retrieval recall |
| Does every cited ID resolve to a chunk the model was shown? | Exact | citation validity |
| Did the assistant refuse to answer an unanswerable question? | Exact (the decision, not the wording) | abstention |
| Is the answer clear, relevant, faithful to its evidence, polite? | Judgment | a rubric, scored by a person or a judge |
Exact tests are cheap and run on every change; use judgment tests only for what exact tests cannot see. This page builds the exact side; Rubrics, Regression, and Release Gates builds the judgment side.
Create a fixed test dataset
A dataset is only useful if it stays the same between runs. We keep two files in kit/data/: the existing rag_cases.jsonl (16 cases, not duplicated) and a new eval_extra.jsonl (6 tool and 5 format cases, one JSON object per line, in the EvalCase shape). A manifest (eval_manifest.json) names the dataset, gives it a version and records a content hash of each file. Loading recomputes the hashes: if a file changed but the manifest did not, you get DatasetDrift. A change then needs a conscious step: new version, new manifest.
# file: kit/evidencedesk/evals.py (add)
# --- 2. A fixed, versioned dataset: a manifest names the files and their content hashes ---
class DatasetDrift(ValueError):
"""A dataset file changed without a new version."""
@dataclass(frozen=True)
class Dataset:
name: str
version: int
sha256: str # short hash over all file hashes
cases: tuple[EvalCase, ...]
def of_kind(self, kind: str) -> list[EvalCase]:
return [c for c in self.cases if c.kind == kind]
def file_hash(path: Path) -> str:
return hashlib.sha256(path.read_bytes().replace(b"\r\n", b"\n")).hexdigest()[:16] # same hash on Windows and Linux
def write_manifest(folder: Path, name: str, version: int, files: dict[str, str]) -> Path:
"""files: {file name: 'rag' | 'eval'}. Run it on purpose, together with a version bump."""
listed = {f: {"format": fmt, "sha256": file_hash(folder / f)} for f, fmt in sorted(files.items())}
path = folder / "eval_manifest.json"
path.write_text(json.dumps({"name": name, "version": version, "files": listed}, indent=1) + "\n", encoding="utf-8")
return path
def load_dataset(manifest: Path, doc_ids: set[str] | None = None) -> Dataset:
meta = json.loads(manifest.read_text(encoding="utf-8"))
cases: list[EvalCase] = []
for fname, info in meta["files"].items():
path = manifest.parent / fname
if (found := file_hash(path)) != info["sha256"]:
raise DatasetDrift(f"{fname} changed (manifest {info['sha256']}, file {found}): bump the version, update the manifest")
if info["format"] == "rag":
cases += [EvalCase.from_rag(c) for c in load_rag_cases(path, doc_ids)]
continue
for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
try:
cases.append(EvalCase.model_validate_json(line))
except ValidationError as error:
raise ValueError(f"{fname} line {number}: {error.errors()[0]['msg']}") from None
if len({c.id for c in cases}) != len(cases):
raise ValueError("case ids must be unique")
combined = "".join(f"{f}:{i['sha256']}" for f, i in meta["files"].items())
return Dataset(meta["name"], meta["version"], hashlib.sha256(combined.encode()).hexdigest()[:16], tuple(cases))The hash ignores Windows line endings, so every machine computes the same value. It detects accidental change; it cannot tell whether the labels are right. Build the dataset (the manifest is rewritten with identical bytes) and trigger the drift error on a copy:
import json
import shutil
from evidencedesk.evals import DatasetDrift, load_dataset, write_manifest
data = Path("kit/data")
files = {"rag_cases.jsonl": "rag", "eval_extra.jsonl": "eval"}
write_manifest(data, "evidencedesk-eval", 1, files) # the same bytes as the committed manifest
dataset = load_dataset(data / "eval_manifest.json")
print(len(dataset.cases), "cases:", {k: len(dataset.of_kind(k)) for k in ("rag", "tool", "format")}, "version", dataset.version)
print({f: i["sha256"][:8] for f, i in json.loads((data / "eval_manifest.json").read_text())["files"].items()})
shutil.copytree(data, "drift")
changed = Path("drift/rag_cases.jsonl")
changed.write_text(changed.read_text(encoding="utf-8").replace("Which frame size", "What frame size"), encoding="utf-8")
try:
load_dataset(Path("drift/eval_manifest.json"))
except DatasetDrift as error:
print("drift:", error)27 cases: {'rag': 16, 'tool': 6, 'format': 5} version 1
{'eval_extra.jsonl': '72ed5a9a', 'rag_cases.jsonl': '3e0cd4d2'}
drift: rag_cases.jsonl changed (manifest 3e0cd4d2a0b6c963, file 59adee656bacd701): bump the version, update the manifestEach tool or format case is one line such as {"id": "tl02", "kind": "tool", "input": {"name": "calculator", "arguments": {...}}, "expect": "invalid_arguments", ..., "reason": "..."}; the file holds all eleven.
Write deterministic unit tests
A deterministic test gives the same verdict on every run and machine, with no network and no model. You can test exactly what earlier pages built: parsing, retrieval, citations, tool validation and fallbacks. The tool for that is the fake client. For suites we add CaseModel: it replies with the hand-written answer for a question if one exists, and otherwise plays the worst case: it copies the top evidence and never abstains. That makes the unanswerable cases a test of your guards, not of the fake's good manners. Tape records what the model said, so we can score it before a guard acts on it.
# file: kit/evidencedesk/evals.py (add)
# --- 3. Fake models for suites (stand-ins: they prove the plumbing, not the quality of a real model) ---
class CaseModel(RecordedReplies):
"""The hand-written reply if there is one; otherwise the WORST case: copy the top evidence, never abstain."""
def complete(self, request: LLMRequest) -> LLMResponse:
question = re.search(r'<question source="user-input">\n(.*?)\n</question>', request.messages[-1].content, re.S)
if question and question.group(1) in self.replies:
return super().complete(request)
self.calls.append(request)
block = re.search(r'<evidence source="([^"]+)">\n(.*?)\n</evidence>', request.messages[-1].content, re.S)
body = reply(block.group(2)[:200], block.group(1)) if block else reply("Not stated.")
return LLMResponse(json.dumps(body), "recorded", Usage(0, 0))
class Tape:
"""Wraps a client and keeps its replies, to score what the model said before any guard acted on it."""
def __init__(self, inner) -> None:
self.inner, self.replies = inner, []
def complete(self, request: LLMRequest) -> LLMResponse:
self.replies.append(self.inner.complete(request))
return self.replies[-1]Most tests you need already exist. Pick the ones that protect each module, so a failure names the module to open:
| Test | Protects |
|---|---|
test_results::test_bad_output_is_rejected_not_raised | parsing (results.py) |
test_citations::test_unresolved_ids_are_split_into_outside_and_invented | citation checks (citations.py) |
test_tools::test_missing_extra_and_invalid_arguments | tool validation (tools.py) |
test_rag::test_missing_evidence_triggers_the_fallback_without_calling_the_model | fallback and retrieval filter (rag.py) |
test_evals::test_a_changed_file_is_drift_until_the_version_is_bumped | the dataset (evals.py) |
test_evals::test_baseline_is_reproducible_and_its_failures_are_the_documented_misses | the whole pipeline |
The kit test file tests/test_evals.py (17 tests) needs the whole module; it is run after the baseline section.
Measure retrieval recall
Retrieval recall asks: is the expected document among the top k results? Per case that is a hit or a miss; aggregated it is the share of hits. Local Indexes and Retrieval Quality introduced recall_at_k. This version runs through the same status filter as production (current chunks only) and uses the evaluation cases. It measures ranking only; the score threshold belongs to the abstention check.
Provider note (as of 2026-10-12): The textbook Introduction to Information Retrieval applies precision and recall to the top k results and calls Precision@k "the least stable of the commonly used evaluation measures". Recall@k is the relevant documents in the top k divided by all relevant documents. MRR (as defined on the NIST TREC QA page) scores each question as 1 divided by the rank of the first correct answer. With one expected document per question, our hit rate equals Recall@k. Check the current documentation.
# file: kit/evidencedesk/evals.py (add)
# --- 4. Metrics on one question or one reply ---
def docs_in_top(retriever: Retriever, question: str, k: int = 3, where: dict | None = None) -> list[str]:
"""Documents of the top-k chunks, best first. where=None -> current chunks only (production); {} -> no filter."""
hits = retriever.index.search(question, retriever.model, k, {"status": "current"} if where is None else where)
return list(dict.fromkeys(retriever.chunks[h.passage_id].doc_id for h in hits))
def retrieval_recall(retriever: Retriever, cases: list[EvalCase], k: int = 3, where: dict | None = None) -> tuple[float, list[str]]:
"""Share of answerable rag cases whose expected document is in the top k, and the ids of the misses."""
asked = [c for c in cases if c.kind == "rag" and c.doc_ids]
missed = [c.id for c in asked if not set(c.doc_ids) & set(docs_in_top(retriever, c.input, k, where))]
return (1 - len(missed) / len(asked) if asked else 1.0), missedfrom evidencedesk.chunking import chunk_corpus
from evidencedesk.evals import docs_in_top, retrieval_recall
from evidencedesk.rag import MIN_SCORE, Retriever
# paragraph chunks, a fixed clock (build_retriever, below, wraps this)
retriever = Retriever(chunk_corpus(Path("kit/corpus")), clock=lambda: "2026-10-13T09:00:00Z", k=3, min_score=MIN_SCORE)
cases = list(dataset.cases)
for k in (1, 3):
value, missed = retrieval_recall(retriever, cases, k)
print(f"recall@{k} = {value:.3f} on 13 answerable cases; missed: {', '.join(missed)}")
for case_id in ("rc06", "rc08"):
case = next(c for c in cases if c.id == case_id)
print(f"{case_id} wanted {case.doc_ids[0]}, top 3 documents: {docs_in_top(retriever, case.input)}")
stale = [c for c in cases if "stale" in c.tags]
for label, where in (("with status filter", None), ("without filter", {})):
print(f"{label:<19} rc09 recall@1 = {retrieval_recall(retriever, stale, 1, where)[0]:.1f}, top document: {docs_in_top(retriever, stale[0].input, 1, where)}")recall@1 = 0.769 on 13 answerable cases; missed: rc02, rc06, rc08
recall@3 = 0.846 on 13 answerable cases; missed: rc06, rc08
rc06 wanted kc-returns-v3, top 3 documents: ['kc-privacy-v1', 'kc-shipping-v1', 'kc-returns-v3-bn']
rc08 wanted kc-giftcards-v1, top 3 documents: ['kc-sizing-v1', 'kc-privacy-v1', 'kc-shipping-v1']
with status filter rc09 recall@1 = 1.0, top document: ['kc-returns-v3']
without filter rc09 recall@1 = 0.0, top document: ['kc-returns-v2']Two cases fail even at k=3: "How do I get my money back?" (no words shared with the returns policy) and the "voucher" question. Their top documents are unrelated, which says what to fix: the embedder, not the prompt. The stale case differs with and without the filter: unfiltered, the superseded policy ranks first and the test fails for a reason production never sees.
Score citation validity
Citation validity is the share of cited IDs that resolve to a chunk the model was actually shown. We reuse check_citations from RAG Reliability and Evaluation Data, which splits an unresolved ID into "real but not shown" and "invented".
# file: kit/evidencedesk/evals.py (add)
def citation_validity(cited: list[str], shown: list[Retrieved], known: set[str]) -> tuple[int, int]:
"""(cited ids that resolve to a chunk the model was shown, distinct cited ids)."""
report = check_citations("", cited, shown, known)
total = len(set(cited))
return total - len(report.outside) - len(report.invented), totalfrom evidencedesk.evals import citation_validity
question = "How long do I have to return a bike?"
shown, known = retriever.retrieve(question), set(retriever.chunks)
for label, cited in (("all resolve", ["kc-returns-v3:c0"]), ("one real but not shown", ["kc-returns-v3:c0", "kc-recalls-v1:c0"]),
("one invented", ["kc-returns-v3:c0", "kc-returns-v9:c0"]), ("no citation", [])):
valid, total = citation_validity(cited, shown, known)
print(f"{label:<24} {valid} of {total}")all resolve 1 of 1
one real but not shown 1 of 2
one invented 1 of 2
no citation 0 of 0The limit from that page still holds: a resolving ID is not support. Validity catches wrong IDs; it does not prove the cited text backs the claim.
Evaluate abstention behavior
Some questions cannot be answered from the corpus; the right behaviour is to say so. Abstention is a decision, so it can be tested exactly, and it fails in two directions: answering the unanswerable (hallucination risk) or abstaining on the answerable (a useless assistant). A 2x2 table shows both. The runner applies four checks to a case (abstention, retrieved, citations_resolve, cites_expected); a case passes only if all pass. run_suite returns plain JSON-able data without timestamps, so a run is reproducible byte for byte.
# file: kit/evidencedesk/evals.py (add)
# --- 5. Running cases ---
@dataclass(frozen=True)
class CaseResult:
id: str
kind: str
checks: dict[str, bool]
detail: str = ""
valid: int = 0 # cited ids that resolved
cited: int = 0 # distinct cited ids
@property
def passed(self) -> bool:
return all(self.checks.values())
def run_rag_case(case: EvalCase, retriever: Retriever, client, k: int = 3, min_margin: float = MIN_MARGIN) -> CaseResult:
tape = Tape(client)
answer = guarded_answer(tape, retriever, case.input, min_margin)
checks = {"abstention": answer.abstained == (case.expect == "abstain")}
if case.doc_ids:
checks["retrieved"] = bool(set(case.doc_ids) & set(docs_in_top(retriever, case.input, k)))
cited: list[str] = []
if tape.replies and (outcome := parse_answer(tape.replies[-1])).ok:
cited = list(outcome.value.cited_ids)
valid, total = citation_validity(cited, retriever.retrieve(case.input), set(retriever.chunks))
if total:
checks["citations_resolve"] = valid == total
if case.must_cite and not answer.abstained:
checks["cites_expected"] = bool(answer.citations) and all(c.doc_id in case.doc_ids for c in answer.citations)
return CaseResult(case.id, "rag", dict(sorted(checks.items())), answer.reason, valid, total)
def run_tool_case(case: EvalCase, registry: ToolRegistry) -> CaseResult:
result = registry.execute(ToolCall(case.id, case.input["name"], case.input["arguments"]))
got = "ok" if result.ok else result.error["code"]
return CaseResult(case.id, "tool", {"outcome": got == case.expect}, got)
def run_format_case(case: EvalCase) -> CaseResult:
reply_ = LLMResponse(case.input["text"], "recorded", Usage(0, 0), case.input.get("stop_reason", "end"))
got = parse_answer(reply_).code
return CaseResult(case.id, "format", {"outcome": got == case.expect}, got)
def build_retriever(corpus: Path, strategy: str = "paragraph", min_score: float = MIN_SCORE, k: int = 3) -> Retriever:
return Retriever(chunk_corpus(corpus, strategy), clock=lambda: "2026-10-13T09:00:00Z", k=k, min_score=min_score)
def _share(done: int, total: int) -> float | None:
return round(done / total, 3) if total else None
def run_suite(dataset: Dataset, retriever: Retriever, client, registry: ToolRegistry, k: int = 3, min_margin: float = MIN_MARGIN,
notes: dict[str, Any] | None = None) -> dict[str, Any]:
"""Run every case. The result is plain JSON-able data: the same input gives the same bytes."""
results = [run_rag_case(c, retriever, client, k, min_margin) if c.kind == "rag" else run_tool_case(c, registry)
if c.kind == "tool" else run_format_case(c) for c in dataset.cases]
rag, answerable = [r for r in results if r.kind == "rag"], {c.id for c in dataset.cases if c.doc_ids}
table = {row: {"answered": 0, "abstained": 0} for row in ("answerable", "unanswerable")}
for r in rag:
table["answerable" if r.id in answerable else "unanswerable"]["answered" if r.detail == "ok" else "abstained"] += 1
passed = lambda kind=None: _share(sum(r.passed for r in results if kind in (None, r.kind)), sum(kind in (None, r.kind) for r in results))
metrics = {"recall_at_k": round(retrieval_recall(retriever, list(dataset.cases), k)[0], 3),
"citation_validity": _share(sum(r.valid for r in rag), sum(r.cited for r in rag)),
"abstention_correct": _share(sum(r.checks["abstention"] for r in rag), len(rag)),
"tool_pass": passed("tool"), "format_pass": passed("format"), "pass_rate": passed()}
config = {"k": k, "min_score": retriever.min_score, "min_margin": min_margin, "chunks": len(retriever.chunks), **(notes or {})}
return {"dataset": {"name": dataset.name, "version": dataset.version, "sha256": dataset.sha256}, "config": config,
"metrics": metrics, "confusion": table,
"cases": {r.id: {"passed": r.passed, "checks": r.checks, "detail": r.detail} for r in results},
"failures": sorted(r.id for r in results if not r.passed)}from evidencedesk.evals import CaseModel, run_suite
from evidencedesk.tools import default_registry
from tests.test_citations import REPLIES # hand-written replies from RAG Reliability and Evaluation Data
registry = default_registry(Path("kit/corpus"))
client = CaseModel({c.input: REPLIES[c.id] for c in dataset.cases if c.id in REPLIES})
result = run_suite(dataset, retriever, client, registry, notes={"chunking": "paragraph"})
print(f"{'':<14}{'answered':>9}{'abstained':>10}")
for row, counts in result["confusion"].items():
print(f"{row:<14}{counts['answered']:>9}{counts['abstained']:>10}")
print("failures:", {i: result["cases"][i]["detail"] for i in result["failures"]})
worst = run_suite(dataset, retriever, CaseModel({}), registry) # a model that never abstains
print("worst-case model, unanswerable:", worst["confusion"]["unanswerable"]) answered abstained
answerable 9 4
unanswerable 0 3
failures: {'rc03': 'weak', 'rc06': 'weak', 'rc08': 'ambiguous', 'rc13': 'ambiguous'}
worst-case model, unanswerable: {'answered': 0, 'abstained': 3}No unanswerable question was answered, even by the worst-case model: the retrieval guard stops them before the model is called. The cost is on the other row: 4 answerable questions were turned away. They are the four known misses of RAG Reliability and Evaluation Data (rc03, rc06, rc08, rc13); the dataset reproduces them on purpose. Safe failure, but still a failure. This is a smoke test: 16 RAG cases show that a rule works or breaks; they cannot rank systems or give a trustworthy percentage, and the replies are hand-written.
Compare outputs against a rubric
Exact checks cannot say whether an answer is good; a person (later a model) rates it against a rubric. The idea for now is one rule: rate relevance, evidence use and format compliance separately. One combined score hides which part failed. Scores holds three ratings (0, 1 or 2) and deliberately has no total.
# file: kit/evidencedesk/evals.py (add)
@dataclass(frozen=True)
class Scores:
"""A manual rubric in miniature: three judgments, each 0 (fails), 1 (partly) or 2 (meets), kept apart on purpose."""
relevance: int
evidence_use: int
format: int
def __post_init__(self) -> None:
if any(v not in (0, 1, 2) for v in asdict(self).values()):
raise ValueError("each score is 0, 1 or 2")from evidencedesk.evals import Scores
rated = { # one human rater, three answers to "How long do I have to return a bike?"
"30 days, cites the returns policy": Scores(relevance=2, evidence_use=2, format=2),
"30 days, no citation": Scores(relevance=2, evidence_use=0, format=2),
"correct text, but not the JSON contract": Scores(relevance=2, evidence_use=2, format=0),
}
for answer, s in rated.items():
print(f"relevance={s.relevance} evidence_use={s.evidence_use} format={s.format} {answer}")relevance=2 evidence_use=2 format=2 30 days, cites the returns policy
relevance=2 evidence_use=0 format=2 30 days, no citation
relevance=2 evidence_use=2 format=0 correct text, but not the JSON contractAnswers two and three both lose points, for different reasons that need different fixes. Writing a proper rubric with observable criteria is the next page's work.
Run a baseline before changing prompts
Before you change a prompt, a chunking rule or a threshold, run the suite and save the results; otherwise you can only say the new numbers look fine. The file records dataset name, version and hash, configuration, metrics and every case's result. Commit it with the code (git add kit/data/baseline_v1.json) and review a change to it like code.
# file: kit/evidencedesk/evals.py (add)
# --- 6. Saved results (the baseline) ---
def save_results(results: dict[str, Any], path: Path) -> None:
path.write_text(json.dumps(results, indent=1, sort_keys=True, ensure_ascii=False) + "\n", encoding="utf-8")
def load_results(path: Path) -> dict[str, Any]:
return json.loads(path.read_text(encoding="utf-8"))from evidencedesk.evals import load_results, save_results
path = Path("kit/data/baseline_v1.json")
save_results(result, path)
saved = load_results(path)
print({k: saved["dataset"][k] for k in ("name", "version")}, saved["config"])
print(saved["metrics"])
rerun = run_suite(dataset, retriever, client, registry, notes={"chunking": "paragraph"})
save_results(rerun, Path("again.json"))
print("second run identical, byte for byte:", Path("again.json").read_bytes() == path.read_bytes()){'name': 'evidencedesk-eval', 'version': 1} {'chunking': 'paragraph', 'chunks': 36, 'k': 3, 'min_margin': 0.05, 'min_score': 0.3}
{'abstention_correct': 0.75, 'citation_validity': 1.0, 'format_pass': 1.0, 'pass_rate': 0.852, 'recall_at_k': 0.846, 'tool_pass': 1.0}
second run identical, byte for byte: TrueThe baseline keeps its four known failures: it is an honest picture of today, and a later change is judged against it.
Now that evals.py is complete, run the kit file tests/test_evals.py (17 tests) together with four earlier files; sed removes pytest's wall-clock seconds:
python -m pytest -q kit/tests/test_evals.py kit/tests/test_rag.py kit/tests/test_tools.py kit/tests/test_results.py kit/tests/test_citations.py | sed 's/ in [0-9.]*s//'........................................................................ [ 88%]
......... [100%]
81 passedExercise: build a small regression suite
Run the same dataset against two retrieval strategies: the default (MIN_SCORE 0.3, margin 0.05) and a lenient one (0.2, no margin check).
from evidencedesk.evals import build_retriever
options = {"default": (0.3, 0.05), "lenient": (0.2, 0.0)} # (min_score, min_margin)
runs = {name: run_suite(dataset, build_retriever(Path("kit/corpus"), min_score=score), client, registry, min_margin=margin,
notes={"strategy": name}) for name, (score, margin) in options.items()}
a, b = runs["default"], runs["lenient"]
for metric in a["metrics"]:
print(f"{metric:<20}{a['metrics'][metric]:>8}{b['metrics'][metric]:>8}")
print("unanswerable answered:", a["confusion"]["unanswerable"]["answered"], "->", b["confusion"]["unanswerable"]["answered"])
print("new failures:", sorted(set(b["failures"]) - set(a["failures"])))recall_at_k 0.846 0.846
citation_validity 1.0 0.875
abstention_correct 0.75 0.688
tool_pass 1.0 1.0
format_pass 1.0 1.0
pass_rate 0.852 0.815
unanswerable answered: 0 -> 3
new failures: ['rc14', 'rc15', 'rc16']The lenient setting answers two more answerable questions, but all three unanswerable ones now get answered by the worst-case model and citation validity drops. Recall is identical, so the problem is invisible if you measure recall alone. Freeze the lesson in pytest. This file is yours (not part of the kit); the lenient case is xfail(strict=True): it documents a known regression and fails loudly if the default ever fails or the regression disappears.
# file: kit/tests/test_lab_regression.py
from pathlib import Path
import pytest
from test_citations import REPLIES
from evidencedesk.evals import CaseModel, build_retriever, load_dataset, run_suite
from evidencedesk.tools import default_registry
CORPUS = Path(__file__).resolve().parent.parent / "corpus"
DATASET = load_dataset(CORPUS.parent / "data" / "eval_manifest.json")
CLIENT = CaseModel({c.input: REPLIES[c.id] for c in DATASET.cases if c.id in REPLIES})
STRATEGIES = {"default": (0.3, 0.05), "lenient": (0.2, 0.0)} # (min_score, min_margin)
KNOWN = pytest.mark.xfail(strict=True, reason="known regression: the lenient setting answers the unanswerable")
def run(name):
score, margin = STRATEGIES[name]
return run_suite(DATASET, build_retriever(CORPUS, min_score=score), CLIENT, default_registry(CORPUS), min_margin=margin)
@pytest.mark.parametrize("name", ["default", pytest.param("lenient", marks=KNOWN)])
def test_critical_unanswerable_questions_are_never_answered(name):
assert run(name)["confusion"]["unanswerable"]["answered"] == 0python -m pytest -q kit/tests/test_lab_regression.py | sed 's/ in [0-9.]*s//'.x [100%]
1 passed, 1 xfailedOne pass and one expected failure: the default holds, and the lenient regression is caught and documented.
Where this shows up at work
Mateo in Bogotá maintains a support assistant. Each change to retrieval settings or prompt goes through a pull request; CI runs about 40 saved questions against a committed baseline, and the reviewer reads the metric changes next to the code diff. The dataset's version and hash show if someone edited the cases to make a number look better. This is an illustration, not a case study.
Common mistakes
- A case without a reason. Nobody knows what its failure means or whether it may be deleted. Make the reason a required field.
- Changing dataset and code in one change. The numbers move for two reasons. Bump the dataset version first, then compare.
- Measuring retrieval without the production filter. Superseded documents rank first and the metric describes a system you do not run.
- One combined quality score. A good format hides bad evidence use. Keep the dimensions apart.
- Reading 16 cases as a benchmark. They find bugs and show direction; a hand-written reply proves the plumbing, not a model.
Try it yourself
- Easy: Revisit Tool Contracts and Execution. Add a tool case that expects the calculator to refuse
2 ** 3(error codebad_expression) to a copy of the dataset. What happens before you update the manifest, and after? - Medium: Revisit Local Indexes and Retrieval Quality. Compare its
recall_at_kwithretrieval_recallon the same labels at k=1 and k=3. Why do they differ at one value of k? - Hard: Revisit RAG Ingestion and Citation Flow (chunking strategies). Run the suite with paragraph chunks and with fixed-size chunks. Which metrics change, which do not, and is the comparison fair?
Answers
Easy. Append a JSON line to a copy, then load: without the manifest update you get DatasetDrift; after write_manifest with version 2 it loads 28 cases.
from evidencedesk.evals import run_tool_case
shutil.copytree(data, "easy")
line = {"id": "tl07", "kind": "tool", "input": {"name": "calculator", "arguments": {"expression": "2 ** 3"}}, "expect": "bad_expression",
"tags": ["tool"], "reason": "Powers could build huge numbers, so the calculator must refuse them."}
with open("easy/eval_extra.jsonl", "a", encoding="utf-8") as f:
f.write(json.dumps(line) + "\n")
try:
load_dataset(Path("easy/eval_manifest.json"))
except DatasetDrift:
print("drift caught: the manifest was not updated")
write_manifest(Path("easy"), "evidencedesk-eval", 2, files)
bigger = load_dataset(Path("easy/eval_manifest.json"))
new = next(c for c in bigger.cases if c.id == "tl07")
print(f"version {bigger.version}, {len(bigger.cases)} cases, tl07:", run_tool_case(new, registry).detail)drift caught: the manifest was not updated
version 2, 28 cases, tl07: bad_expressionPass checklist: drift caught; version raised; the new case has a reason and passes.
Medium. The kit's recall_at_k searches all chunks; ours searches current chunks only.
from evidencedesk.index import recall_at_k
labels = [(c.input, c.doc_ids[0]) for c in cases if c.kind == "rag" and c.doc_ids]
for k in (1, 3):
plain, _ = recall_at_k(retriever.index, retriever.model, labels, k)
print(f"k={k}: recall_at_k (no filter) {plain:.3f}, retrieval_recall (current only) {retrieval_recall(retriever, cases, k)[0]:.3f}")k=1: recall_at_k (no filter) 0.692, retrieval_recall (current only) 0.769
k=3: recall_at_k (no filter) 0.846, retrieval_recall (current only) 0.846They differ at k=1 only: unfiltered, the superseded policy ranks first for the stale question; at k=3 the current one is also in the list. Pass checklist: both numbers for both k; the stale case named as the reason.
Hard. Change only the chunking strategy:
for strategy in ("paragraph", "fixed"):
r = build_retriever(Path("kit/corpus"), strategy)
res = run_suite(dataset, r, client, registry, notes={"chunking": strategy})
m = res["metrics"]
print(f"{strategy:<10} chunks={res['config']['chunks']} recall={m['recall_at_k']} citation_validity={m['citation_validity']} pass_rate={m['pass_rate']}")paragraph chunks=36 recall=0.846 citation_validity=1.0 pass_rate=0.852
fixed chunks=38 recall=0.846 citation_validity=0.889 pass_rate=0.815Document-level recall is the same, but citation validity drops and the pass rate with it: exactly one case changes, rc09, with reason unknown_citation. Its hand-written reply cites the paragraph chunk kc-returns-v3:c2, which the fixed-size strategy does not retrieve. The test is tied to one chunking, so this comparison mixes the strategy with the fake's chunk IDs. A fair comparison needs replies per strategy (or a real model). Pass checklist: only chunking changed; recall unchanged; the one new failure traced to its cause.
Summary
- A case stores input, expected behaviour and a reason; a versioned, hashed dataset turns drift into an error.
- Exact tests cover parsing, ids, tools, retrieval, citations and abstention; judgment goes into a rubric, scored per dimension.
- Measure through the production path: recall with the status filter, validity against what the model saw, abstention as a 2x2 table.
- Save a baseline, failures included, before changing anything; compare strategies on the same dataset in pytest.
- Sixteen cases and a fake model prove plumbing and direction, not quality.
Next: Rubrics, Regression, and Release Gates turns judgment into an explicit rubric, compares outputs side by side, and sets a gate that fails the build when a critical check regresses.