Reranking and Retrieval Evaluation

The previous page ended with hit rates: how many of 21 questions had the answer in the top 1, 3 or 5. That is a blunt number. It does not say where the answer sits, it ignores partly useful evidence, and it cannot tell whether a gap of one question means anything. Aiko in Osaka is asked "does the new reranker help?" and can be wrong in two ways: yes because a number rose, or no because it fell. This page gives her the tools to say "this much, and this unsure".

It adds a cross-encoder-shaped reranker, the four standard retrieval metrics and the honesty tools that go with them, then sizing, duplicates, freshness, tenant boundaries, citation checks, ablations and a runnable benchmark. All new code is in evidencedesk/retrieval/ and reuses the previous page's catalog, labels, searchers and TwoStage. Two warnings. "Dense" still means the toy embedder of Embeddings and Similarity. And no real cross-encoder runs here: the rerankers are deterministic stand-ins that read words, not meaning. The numbers show the method on 24 questions, not what a production model would do.

What you will learn

• How a cross-encoder scores a (query, passage) pair, why its score is not a probability, and how to test a reranker offline.
• Recall@k, precision@k, MRR and nDCG: exact definitions, which variant this course uses, and what each cannot see.
• How to report a difference honestly with 24 labels: intervals, per-question flips, ties and slices.
• How candidate count, final context size, duplicates, freshness and tenant filters change quality and cost.
• How to run ablations, feed a retrieval result into the release gate, and write a recommendation.

Use a cross-encoder reranker

A bi-encoder (the toy of Embeddings and Similarity) embeds query and passage separately and compares vectors. A cross-encoder reads the query and one passage together and outputs one number for the pair. The price: one model run per (query, candidate) pair, nothing precomputed. So it sits in stage two of TwoStage: the cheap first stage cuts the corpus to a few dozen candidates, the expensive stage orders only those.

Provider note (as of 2026-10-12): the sentence-transformers documentation shows retrieve-and-rerank: a first stage fetches e.g. 100 candidates, a CrossEncoder re-scores each pair, you sort by score. It calls a cross-encoder the wrong tool for searching a whole corpus (its example: 10,000 sentences, about 50 million pairs, about 65 hours, against about 5 seconds with a bi-encoder; hardware-dependent). The docs call the output a score "between 0 and 1", but the model card of cross-encoder/ms-marco-MiniLM-L6-v2 prints raw scores such as 8.607138 and -4.320078: a ranking score, never a probability. That model is Apache-2.0, has 22.7M parameters, is trained on English data and downloads its weights on first load; its Bangla quality is not shown (unverified). Check the current documentation.

The kit defines the interface and ships stand-ins. PairScorer is a protocol with one method, predict(pairs): a list of (query, passage) pairs in, one float per pair out, named like the library call. PairReranker adapts any such scorer to the Reranker protocol, counts the pairs it scores (the cost) and can cut passages to max_words (real models read a limited number of tokens).

LexicalPairScorer is the stand-in: four hand-weighted features (query-word coverage, adjacent word pairs, exact identifiers such as RD2-24, title match) mapped to 10 * weighted sum - 4, so raw scores fall outside 0 to 1 as a real model's do. It cannot read a paraphrase. Score six candidates of the recall question:

from evidencedesk.retrieval.catalog import Scope, load_catalog
from evidencedesk.retrieval.labels import load_labels
from evidencedesk.retrieval.lifecycle import build_searchers
from evidencedesk.retrieval.rerankers import LexicalPairScorer

cat, labels = load_catalog("paragraph"), load_labels()
parts, allowed = build_searchers(cat), cat.allowed(Scope())
q = "Is my bike affected by RD2-24?"
top = parts["hybrid"].search(q, 20, allowed)[:6]  # stage one: the six best of twenty candidates
scorer = LexicalPairScorer()  # stage two: one score per (query, passage) pair
raw = scorer.predict([(q, f"{c.chunk.title} || {c.chunk.text}") for c in top])
for c, s in sorted(zip(top, raw), key=lambda pair: -pair[1]):
    print(f"first-stage rank {c.rank}  raw score {s:>6}  {c.chunk.id}")

first-stage rank 1  raw score  9.333  kc-recalls-v1:c0
first-stage rank 4  raw score  0.167  kc-assembly-v1:c2
first-stage rank 2  raw score -0.667  kc-returns-v3:c0
first-stage rank 3  raw score -0.667  kc-service-v1:c1
first-stage rank 5  raw score -3.167  kc-sizing-v1:c2
first-stage rank 6  raw score -3.167  kc-assembly-v1:c1

A 9.3 beside negative numbers is no probability. Ranks 2 and 3 tie at -0.667 and keep first-stage order: a tie-break, not a judgement (ties_at_top counts them). The real scorer is the same few lines behind the same interface; it has no output because it needs a download:

# not run here: the first load downloads model weights (needs network, disk and the sentence-transformers package)
from sentence_transformers import CrossEncoder


class HubPairScorer:
    # same predict() shape as LexicalPairScorer: PairReranker(HubPairScorer()) is a one-line swap
    def __init__(self, name: str = "cross-encoder/ms-marco-MiniLM-L6-v2") -> None:
        self.model = CrossEncoder(name)

    def predict(self, pairs):
        return [float(s) for s in self.model.predict(list(pairs))]  # raw scores such as 8.6 or -4.3

Measure recall@k, precision@k and MRR

All three start from the labels of the previous page: a chunk is relevant if its grade is 2 (it answers the question). For one question, with R relevant chunks in the allowed set:

MRR follows the NIST TREC question-answering page. Precision and recall at k are the set measures applied to the top k; the IR book calls precision@k the least stable common measure. A short list still divides by k. Questions with no relevant chunk (the three unanswerable ones) are undefined, not zero: they are left out. Check the code on a hand-worked list first:

from evidencedesk.retrieval.metrics import Judged, ndcg_at, precision_at, recall_at, reciprocal_rank

# One question. Two chunks answer it (grade 2, say the FAQ copy and the policy); one more is useful context (grade 1).
# Our list has five slots: [useless, answer, context, useless, answer]. `pool` = grades of ALL allowed chunks, best first.
j = Judged("demo", ("direct",), True, grades=(0, 2, 1, 0, 2), pool=(2, 2, 1))
for k in (1, 2, 5):
    print(f"k={k}  recall {recall_at(j, k):.3f}  precision {precision_at(j, k):.3f}")
print(f"reciprocal rank {reciprocal_rank(j):.3f}  (first answer at rank 2)")
print(f"ndcg@5 exponential gain {ndcg_at(j, 5):.3f}   linear gain {ndcg_at(j, 5, 'linear'):.3f}")
print(f"the same list with the two answers first: {ndcg_at(Judged('d', (), True, (2, 2, 1, 0, 0), j.pool), 5):.3f}")

k=1  recall 0.000  precision 0.000
k=2  recall 0.500  precision 0.500
k=5  recall 1.000  precision 0.400
reciprocal rank 0.500  (first answer at rank 2)
ndcg@5 exponential gain 0.659   linear gain 0.674
the same list with the two answers first: 1.000

Recall reaches 1 as the second copy enters; precision is capped by R/k; the reciprocal rank stops at the first answer. Now the three strategies of the previous page, ten results per question:

from evidencedesk.retrieval.metrics import judge_run, summarize

names = ["recall@1", "recall@3", "recall@5", "precision@3", "mrr", "ndcg@5", "ndcg@5:linear"]
print(f"{'':<8}" + "".join(f"{n:>14}" for n in names))
runs = {}
for name in ("sparse", "dense", "hybrid"):
    runs[name] = judge_run(parts[name], cat, labels, Scope(), 10)
    print(f"{name:<8}" + "".join(f"{v:>14.3f}" for v in summarize(runs[name], names).values()))

              recall@1      recall@3      recall@5   precision@3           mrr        ndcg@5 ndcg@5:linear
sparse           0.714         0.905         0.952         0.302         0.813         0.840         0.837
dense            0.667         0.857         0.905         0.286         0.756         0.779         0.774
hybrid           0.762         0.905         0.952         0.302         0.835         0.846         0.839

The order matches the previous page, and the new columns add information. precision@3 is 0.302 against a ceiling of 0.333, because nearly every question has one answering chunk: a property of the labels, not a defect. MRR puts hybrid 0.022 above sparse; the next sections put that in perspective.

Measure nDCG on graded relevance labels

Some evidence is better than other evidence. rc02 ("Is my bike affected by RD2-24?") has a grade-2 chunk (the recall notice) and a grade-1 chunk (where the serial number is). nDCG rewards grade 2 above grade 1 above nothing and divides by the best possible score, so a perfect list gets 1.0. The textbook form (IR book, equation 44) uses gain 2^grade - 1 and discount log2(1 + rank); this course uses exactly that, and variant="linear" (gain = grade) is the other common form. Libraries differ and one evaluator's documentation does not say which it uses: check before comparing an nDCG across tools.

# file: evidencedesk/retrieval/metrics.py (excerpt)
def gain(grade: int, variant: str = "exp") -> float:
    if variant == "exp":
        return 2.0**grade - 1  # the textbook form: grade 1 -> 1, grade 2 -> 3
    if variant == "linear":
        return float(grade)  # the other common form: grade 1 -> 1, grade 2 -> 2
    raise ValueError("variant must be 'exp' or 'linear'")


def dcg(grades: Iterable[int], variant: str = "exp") -> float:
    """sum of gain(grade) / log2(1 + rank), ranks counted from 1."""
    return sum(gain(g, variant) / math.log2(1 + n) for n, g in enumerate(grades, 1))


def ndcg_at(j: Judged, k: int, variant: str = "exp") -> float | None:
    """DCG of the list / DCG of the best possible list. 1.0 = nothing better could be shown. Libraries differ in `variant`."""
    ideal = dcg(j.pool[:k], variant)
    return dcg(j.grades[:k], variant) / ideal if ideal else None

The ideal list comes from every allowed chunk (pool), not from what search returned, so a missing grade-2 chunk costs even when nothing else is wrong; and it depends on the scope. In the hand-worked example the variants give 0.659 and 0.674, and 1.000 once the answers lead. In the strategy table they differ in the third decimal at most, typical when most grades are 0 or 2.

What a metric can and cannot support

With 21 answerable questions one question is worth 4.8 points of recall@1. The kit gives three tools: a seeded bootstrap interval (resample the questions), flip counts (questions where B beats or loses to A, the only ones carrying information) with an exact sign-test p, and slices by tag with sizes.

from evidencedesk.retrieval.metrics import bootstrap_ci, by_tag, paired, per_question

mrr = {n: per_question(runs[n], "mrr") for n in runs}
for n, values in mrr.items():
    lo, hi = bootstrap_ci(list(values.values()), seed=0)
    print(f"{n:<7} MRR {sum(values.values()) / len(values):.3f}   95% interval [{lo}, {hi}]   ({len(values)} questions)")
d = paired(mrr["sparse"], mrr["hybrid"])
print(f"hybrid vs sparse: better on {list(d.better)}, worse on {list(d.worse)}, same on {d.tied}; sign-test p = {d.p}")
print("hybrid MRR by tag (value/questions):", ", ".join(f"{t} {v:.2f}/{n}" for t, (v, n) in by_tag(runs["hybrid"], "mrr").items()))

sparse  MRR 0.813   95% interval [0.683, 0.94]   (21 questions)
dense   MRR 0.756   95% interval [0.59, 0.905]   (21 questions)
hybrid  MRR 0.835   95% interval [0.686, 0.952]   (21 questions)
hybrid vs sparse: better on ['rc05', 'rc07', 'rc10'], worse on ['h01', 'h02', 'h03'], same on 15; sign-test p = 1.0
hybrid MRR by tag (value/questions): bangla 1.00/3, direct 1.00/2, exact_term 0.83/4, paraphrase 0.75/6, restricted 1.00/1, stale 1.00/1, structured 0.67/6

The intervals overlap almost completely. Hybrid beats sparse on three questions and loses three others: a wash, p = 1.0, though the means differ. The pattern is the useful part: hybrid wins paraphrase and Bangla questions (rc05, rc07, rc10) and loses three structured-document questions. Read slices with their size: "structured 0.67" rests on six questions, "restricted 1.00" on one; a two-tag question sits in two slices. Rules: a one- or two-question gap is noise; report interval and flips; never pick the winner of the third decimal.

Tune candidate count and final context size

Two settings shape a two-stage pipeline: candidates (fetched by stage one, scored by stage two) and final (chunks reaching the prompt). More candidates raise the chance the answer is in the pool and cost grows linearly; a larger final size raises recall and adds tokens and distraction. The sweep uses the word-overlap reranker of the previous page. Latency is a hypothetical price list (4 ms for stage one plus 1.5 ms per scored pair); the kit counts pairs and tokens and never times anything.

from evidencedesk.retrieval.evalreport import Config, Costs, run_config

print("candidates  final  recall@final  mrr  pairs/question  context tokens  est. ms (hypothetical)")
for depth, final in [(3, 3), (3, 5), (10, 1), (10, 3), (10, 5), (20, 3)]:
    r = run_config(Config(reranker="overlap", depth=depth, final=final), parts, cat, labels)
    print(f"{depth:>10}{final:>7}{r.summary['recall@' + str(final)]:>14.3f}{r.summary['mrr']:>7.3f}{r.scored_per_q:>16.1f}{r.tokens_per_q:>16.0f}{r.latency_ms(Costs(4.0, 1.5)):>12}")

candidates  final  recall@final  mrr  pairs/question  context tokens  est. ms (hypothetical)
         3      3         0.905  0.849             3.0              97         8.5
         3      5         0.905  0.849             3.0              97         8.5
        10      1         0.857  0.857            10.0              36        19.0
        10      3         0.952  0.897            10.0             105        19.0
        10      5         0.952  0.897            10.0             173        19.0
        20      3         0.952  0.897            20.0             105        34.0

With 3 candidates the pool is the limit and recall stops at 0.905. Ten candidates buy +0.047 recall@3 for 7 more pairs; 20 buy nothing and raise the hypothetical latency from 19 to 34 ms. Here 10 is the knee, specific to 24 labels and 52 chunks. Final 1 saves tokens (36 against 105) and gives up 0.095 recall. Choose the smallest setting inside the noise of the best, and re-measure when the corpus grows.

When reranking hurts

A reranker costs pairs and time, can only reorder what stage one fetched, and can discard what stage one knew. The ablation keeps the stand-in as the pipeline under test and changes one setting per row (see the ablation section):

from evidencedesk.retrieval.evalreport import ablate

changes = [("reranker", "none"), ("reranker", "overlap"), ("reranker", "blend"), ("max_words", 8),
           ("depth", 3), ("prefilter", False)]
print(f"{'change (one at a time)':<22}{'mrr':>7}{'delta':>8}  better/worse  p      pairs/q")
for row in ablate(Config(reranker="pair", final=5), changes, parts, cat, labels):
    print(f"{row['change']:<22}{row['run'].summary['mrr']:>7.3f}{row['delta']:>+8.3f}  {len(row['better'])}/{len(row['worse']):<11} {row['p']:<6} {row['run'].scored_per_q:>6.1f}")

change (one at a time)    mrr   delta  better/worse  p      pairs/q
(base)                  0.817  +0.000  0/0           1.0      20.0
reranker=none           0.835  +0.018  4/2           0.688     0.0
reranker=overlap        0.897  +0.080  4/0           0.125    20.0
reranker=blend          0.841  +0.024  3/2           1.0      20.0
max_words=8             0.770  -0.047  1/2           1.0      20.0
depth=3                 0.802  -0.015  2/1           1.0       3.0
prefilter=False         0.738  -0.079  0/3           0.25     20.0

Removing the reranker raises MRR (+0.018): the stand-in reads words only and demotes the paraphrase and Bangla questions that the dense half had ranked first. Cutting passages to 8 words drops MRR by 0.047: RD2-24 is the eleventh word of the recall notice, so it is cut before scoring; a real model has the same limit in tokens. Blending the first-stage order with the reranker's by RRF recovers most of the loss, a cheap hedge. And the p column: even the best change, the overlap reranker (+0.080, better on 4, worse on 0), has p = 0.125. Nothing here is distinguishable from noise.

Detect duplicate and near-duplicate evidence

Documents get copied: a policy page and an FAQ share a sentence. Duplicates waste prompt slots and push distinct evidence out of the top k. Two new data files add a FAQ copy of the 30-day rule and a partner page with the same words but 14 days. dedupe uses the Beginner word-overlap measure (Context Quality, Budgets, and Reproducibility) plus one rule: chunks whose numbers differ are different claims and both stay. The block also shapes the rest of the evidence: a token budget (fit_tokens), reading order with the strongest chunks at the edges (edges_first) and sentence-level compression with quoted spans (pick_spans).

from evidencedesk.retrieval.catalog import CORPUS, DOCS
from evidencedesk.retrieval.evidence import context_tokens, dedupe, edges_first, fit_tokens, keeps_evidence, pick_spans
from evidencedesk.retrieval.labels import DATA
from evidencedesk.retrieval.searchers import HybridSearcher

dup_cat = load_catalog("paragraph", folders=(CORPUS, DOCS, DATA / "dups"))  # adds a FAQ copy and a 14-day partner page
hyb = HybridSearcher(*[build_searchers(dup_cat)[n] for n in ("sparse", "dense")])
q = "How long do I have to return a bike?"
found = hyb.search(q, 5, dup_cat.allowed(Scope()))
print("before:", [c.chunk.id for c in found])
d = dedupe(found)
print("after: ", [c.chunk.id for c in d.kept])
print("merged:", d.merged, "| same words, different number, kept apart:", d.held_back)
print("reading order (ranks):", [c.rank for c in edges_first(d.kept)])
print("tokens kept under a 100-token budget:", context_tokens(fit_tokens(d.kept, 100)[0]), "of", context_tokens(d.kept))
lost, before, after = [], 0, 0
for x in labels:  # sentence-level compression on the normal catalog, three chunks per question
    if x.answerable:
        top = parts["hybrid"].search(x.question, 3, allowed)
        spans = pick_spans(x.question, top)
        before, after = before + context_tokens(top), after + context_tokens(spans)
        if any(x.grade_of(c.chunk) == 2 for c in top) and not keeps_evidence(x, spans):
            lost.append(x.id)
print(f"best sentence per chunk: {before} -> {after} tokens over 21 questions; answer cut off for {lost}")

before: ['kc-returns-faq-v1:c0', 'kc-returns-partner-v1:c0', 'kc-returns-v3:c0', 'kc-warranty-v2:c0', 'kc-sizing-v1:c1']
after:  ['kc-returns-faq-v1:c0', 'kc-returns-partner-v1:c0', 'kc-warranty-v2:c0', 'kc-sizing-v1:c1']
merged: {'kc-returns-faq-v1:c0': ('kc-returns-v3:c0',)} | same words, different number, kept apart: (('kc-returns-partner-v1:c0', 'kc-returns-faq-v1:c0'),)
reading order (ranks): [1, 3, 4, 2]
tokens kept under a 100-token budget: 99 of 186
best sentence per chunk: 2057 -> 1148 tokens over 21 questions; answer cut off for ['rc12', 'h06', 'h07']

The policy copy is merged into the FAQ chunk (merged still names it, so both sources can be listed) and the 14-day page survives; whether it belongs in the prompt is a conflict to resolve, as in Context Quality, Budgets, and Reproducibility. The labels count both copies as answers, so dedupe lowers recall@k on paper while giving the model the same evidence: judge it by hit rate and MRR. Reading order is a hypothesis to test with your model, not a result. Compression is the cautionary tale: the best sentence per chunk cuts the prompt from 2057 to 1148 estimated tokens and cuts the answer off for three questions (rc12, h06, h07). Measure before using it.

Handle freshness and deletions

A document changes (30 days becomes 45) or must go. Replace adds the new version and marks older ones superseded (Beginner update_document): kept for audit, hidden by a scope filter. Delete removes every chunk, for a retention or legal request. Neither works unless the index follows. Three states:

from dataclasses import replace

from evidencedesk.chunking import chunk_document, load_document
from evidencedesk.retrieval.catalog import CORPUS
from evidencedesk.retrieval.lifecycle import delete_document, replace_document, served

doc = load_document(CORPUS / "returns-policy.md")  # the policy changes: 30 days becomes 45
v4 = [replace(c, id=c.id.replace("v3", "v4"), doc_id="kc-returns-v4", version=4, text=c.text.replace("30 days", "45 days")) for c in chunk_document(doc)]
cat2, change = replace_document(cat, v4)
print("superseded:", list(change.superseded))
top2 = lambda found: [c.chunk.id for c in found][:2]
print("old index, no filter:      ", top2(parts["hybrid"].search(q, 3, None)))
print("old index, new filter:     ", top2(parts["hybrid"].search(q, 3, cat2.allowed(Scope()))), "<- answer gone")
print("rebuilt index, new filter: ", top2(build_searchers(cat2)["hybrid"].search(q, 3, cat2.allowed(Scope()))))
_, gone = delete_document(cat2, "kc-returns-v2")  # a retention rule: delete, do not just hide
print("deleted", list(gone.removed), "- old index still serves:", sorted(served(parts["sparse"], ["store credit"], None, 20) & set(gone.removed)))

superseded: ['kc-returns-v3:c0', 'kc-returns-v3:c1', 'kc-returns-v3:c2']
old index, no filter:       ['kc-returns-v2:c0', 'kc-returns-v3:c0']
old index, new filter:      ['kc-warranty-v2:c0', 'kc-sizing-v1:c1'] <- answer gone
rebuilt index, new filter:  ['kc-returns-v4:c0', 'kc-warranty-v2:c0']
deleted ['kc-returns-v2:c0', 'kc-returns-v2:c1'] - old index still serves: ['kc-returns-v2:c1']

The old index without a filter still serves the superseded policy. With the new filter it hides the stale text and loses the answer, because the new document was never embedded. Only a rebuild (or an upsert) is right. After a delete the old index still holds the text: a filter is a safety net, not a deletion. Write down what triggers re-indexing, how long old chunks stay visible, and a test that asks the old question after every change. Retrieval Operations and Data Freshness turns this into operations.

Design multi-tenant retrieval boundaries

Two companies share one index, and a Harbor Bikes question must never return a Kestrel chunk, even for an exact identifier typed on purpose. The boundary is the allowed set: the scope's rules and the tenant's documents. Design points: a document with no owner is returned to nobody (fail closed); an unknown tenant gets an empty set, which every searcher treats as "nothing", never as "no filter" (None); the filter runs before ranking. Then ask every question as every tenant:

from evidencedesk.retrieval.lifecycle import TENANT_DOCS, Tenancy, boundary_leaks

full = load_catalog("paragraph", folders=(CORPUS, DOCS, TENANT_DOCS))  # adds two documents of a second company
tenancy, searchers = Tenancy.load(), build_searchers(full)
asks = [x.question for x in labels] + ["HB-77", "Harbor Bikes returns 60 days", "RD2-24", "store credit"]
print("documents with no owner:", tenancy.unowned(full), "| leaks with the tenant filter:", len(boundary_leaks(searchers, full, tenancy, asks, Scope())))
for t in ("kestrel", "harbor", "nobody"):
    top = searchers["hybrid"].search(q, 1, tenancy.allowed(full, t, Scope()))
    print(f"{t:<8} -> {top[0].chunk.id if top else 'nothing'}")
unfiltered = {c.chunk.id for a in asks for s in searchers.values() for c in s.search(a, 5, full.allowed(Scope())) if c.chunk.doc_id.startswith("hb-")}
print("without the tenant filter the same questions return:", sorted(unfiltered))

documents with no owner: [] | leaks with the tenant filter: 0
kestrel  -> kc-returns-v3:c0
harbor   -> hb-returns-v1:c0
nobody   -> nothing
without the tenant filter the same questions return: ['hb-recall-v1:c0', 'hb-returns-v1:c0']

No leaks for 28 questions, three searchers and both tenants, and each tenant gets its own answer. The last line is the control: unfiltered, the same questions do return the other company's chunks. A leak test without a control may simply never have asked a dangerous question. This tests the retrieval layer; who the tenant is is the application's job (Prompt Injection and Access Boundaries), developed in Ingestion and Permission-Aware Retrieval.

Evaluate citation coverage and support

Retrieval ends in an answer that cites. Three rates per answer, with inline [chunk id] markers: coverage (sentences that cite anything), resolve (citations naming a chunk the model was shown) and support (cited sentences whose words and numbers appear in the cited text: Beginner check_support). Six hand-written answers, one case each:

import json

from evidencedesk.retrieval.evidence import check_sentences, citation_scores

text = {c.id: c.text for c in cat.chunks}
print(f"{'case':<6}{'coverage':>9}{'resolve':>9}{'support':>9}  what the case tests")
for line in (DATA / "answers.jsonl").read_text(encoding="utf-8").splitlines():
    a = json.loads(line)
    s = citation_scores(check_sentences(a["answer"], {i: text[i] for i in a["shown"]}))
    print(f"{a['id']:<6}{s['coverage']:>9}{s['resolve']:>9}{s['support']:>9}  {a['note']}")

case   coverage  resolve  support  what the case tests
ans1        1.0      1.0      1.0  Every sentence cites a shown chunk that holds its words and numbers.
ans2        0.5      1.0      1.0  The second sentence makes a claim and cites nothing.
ans3        1.0      0.0      0.0  A real chunk id, but the model was not shown that chunk.
ans4        1.0      1.0      0.0  The id resolves, the words match, the number is wrong.
ans5        1.0      1.0      1.0  The cited text says opened helmets CANNOT be returned. Words and numbers all match.
ans6        1.0      1.0      0.5  One sentence is supported; the other cites the wrong chunk and contradicts the right one.

Each failure shows in a different column. ans5 is the limit: it scores perfectly and is wrong, because the cited text says opened helmets cannot be returned and the check compares words and numbers only. A resolving, "supported" citation is not proof of support. Negation needs a human or an entailment judge calibrated against human labels (Evaluation Design and Judge Calibration), and the number check is set-based: a "4" anywhere in the chunk passes.

Run ablation experiments

An ablation changes one component at a time so you know which part earned its cost. Config holds every choice in one frozen dataclass; ablate runs the base and one row per single-field change, reporting metrics, the MRR change, better/worse counts, p and cost (the table in the "reranking hurts" section is exactly this). Report a row as a tuple, not a verdict: "pre-filter off: MRR -0.079, worse on 3, better on 0, p = 0.25", the clearest loss in the table, as the previous page predicted.

Mini-lab: a hybrid RAG benchmark

The benchmark runs seven variants on the same chunks, filter and labels and prints one table. Everything is deterministic except the bootstrap (--seed). "vs first" counts questions whose reciprocal rank is higher (+) or lower (-) than in the first row. Run it, then this page's five test files:

python -m evidencedesk.retrieval.benchmark --gate

variant                  r@1   r@3    mrr  95% interval   ndcg@5  vs first pairs/q
sparse                 0.714 0.905  0.813  [0.68, 0.94]     0.840         -     0.0
dense                  0.667 0.857  0.756  [0.59, 0.91]     0.779     +3/-6     0.0
hybrid                 0.762 0.905  0.835  [0.69, 0.95]     0.846     +3/-3     0.0
hybrid+overlap         0.857 0.952  0.897  [0.76, 1.00]     0.893     +4/-1    20.0
hybrid+feature         0.714 0.952  0.817  [0.68, 0.94]     0.837     +2/-2    20.0
hybrid+blend           0.762 0.952  0.841  [0.70, 0.95]     0.849     +4/-2    20.0
hybrid+pair (8 words)  0.667 0.905  0.770  [0.62, 0.91]     0.805     +3/-4    20.0
GATE: GO

python -m pytest tests/test_retrieval_metrics.py tests/test_retrieval_rerankers.py tests/test_retrieval_evidence.py \
  tests/test_retrieval_lifecycle.py tests/test_retrieval_evalreport.py 2>&1 | tail -1 | sed 's/ in [0-9.]*s//'

58 passed

The last line is the retrieval release gate. Rubrics, Regression, and Release Gates built gate.py: policy file, saved baseline, critical cases, block and review verdicts. gate_result shapes a retrieval run like the results that gate reads (recall_at_k, pass_rate, one case per answerable question); gate_policy_retrieval.json names four critical questions (rc02, rc03, rc09, h08: identifier, stale policy, restricted note). That gate cannot see ranking quality, so retrieval_gate adds a review line when MRR got worse for more questions than it got better. Four changes against the saved baseline (hybrid, three chunks, no reranker):

from evidencedesk.gate import GatePolicy
from evidencedesk.retrieval.evalreport import retrieval_gate

policy = GatePolicy.model_validate_json((DATA / "gate_policy_retrieval.json").read_text(encoding="utf-8"))
base = run_config(Config(), parts, cat, labels)  # the saved baseline: hybrid, no reranker, three chunks
for name, cfg in [("rerank overlap", Config(reranker="overlap")), ("rerank feature", Config(reranker="feature")),
                  ("rerank, 8-word cut", Config(reranker="pair", max_words=8)), ("no pre-filter", Config(prefilter=False))]:
    print(f"== {name}\n" + retrieval_gate(policy, run_config(cfg, parts, cat, labels), base).text())

== rerank overlap
GATE: GO
== rerank feature
GATE: REVIEW
  review: mrr worse for 4 question(s) ['h06', 'rc05', 'rc07', 'rc10'], better for 2 (sign-test p=0.688)
== rerank, 8-word cut
GATE: BLOCK
  block:  critical case rc02 failed its retrieved check
  review: mrr worse for 4 question(s) ['h06', 'rc02', 'rc05', 'rc07'], better for 1 (sign-test p=0.375)
== no pre-filter
GATE: REVIEW
  review: mrr worse for 3 question(s) ['h08', 'rc01', 'rc09'], better for 0 (sign-test p=0.25)

The overlap reranker is a clean GO. The feature reranker passes the Beginner thresholds (recall is higher) yet gets a review line: four questions got worse (three paraphrase or Bangla, one table), and p = 0.688 says it may be noise, so it asks a person instead of blocking. The 8-word cut is a BLOCK because a critical identifier question fails, whatever the averages say. Dropping the pre-filter is only a review on these labels, yet it is the change a human most needs to see. A gate input is a signal, not a verdict.

My recommendation (write yours first). Keep hybrid with the pre-filter as the first stage; do not ship the feature or truncated rerankers. The overlap reranker is the best candidate (MRR 0.897 against 0.835), but the intervals overlap and 4 to 0 is p = 0.125: encouraging, not proven. Before deciding, add labelled questions (the Hard task counts how many), test a real cross-encoder on the same file, and count its latency at the depth you can afford. No ranking method makes the system abstain on unanswerable questions; that check sits outside retrieval.

New in the kit: modules metrics, rerankers, evidence, lifecycle, evalreport, benchmark; data answers.jsonl, tenants.json, gate_policy_retrieval.json, baseline_retrieval.json, folders dups/, tenants/; five tests/test_retrieval_*.py files.

Where this shows up at work

A search team ships a reranker after a demo where five queries improved; next week support reports missed identifier questions. A labelled set with flip counts would have shown it in review: "better on 4, worse on 3, p = 1.0" is a different sentence from "MRR up 0.02". The same table feeds capacity planning (pairs per question times a measured cost per pair), the on-call question "did this release change retrieval?" (the saved baseline) and the review of a shared index (the leak test with its control).

Common mistakes

• Comparing means without flips. "0.835 against 0.813" is two questions. Report the interval and the better/worse counts.
• Mixing nDCG variants. Gain 2^grade - 1 and linear gain differ, and so do libraries. State the variant beside every number.
• Reading a reranker score as confidence. A raw 8.6 is not 86%, and scores of different queries are not comparable. Use it for order only.
• Tuning on the labels, then quoting them. Choosing depth, weights or reranker on the 21 questions and reporting the same 21 is optimistic. Hold some out, or label more.
• A leak test without a control, or a metric that punishes the fix. Show that the unfiltered run does leak; judge dedupe by hit rate, since recall counts each copy.

Try it yourself

1. Easy (revisits Local Indexes and Retrieval Quality): a question has two answering chunks. The list has grades (0, 0, 2). Predict recall@3, precision@3 and the reciprocal rank by hand, then check with Judged("x", (), True, (0, 0, 2), (2, 2)).
2. Medium (revisits RAG Reliability and Evaluation Data): write an answer, in the format of answers.jsonl, that is wrong about delivery time but scores coverage, resolve and support of 1.0. Explain which check would catch it.
3. Hard (revisits Rubrics, Regression, and Release Gates): how many questions must flip in the same direction, with none flipping back, before the sign test gives p below 0.05? What share of this label set is that, and what does it mean for a release gate?

Answers

Easy. One of two relevant chunks is found: recall 1/2 = 0.5. One of three slots is relevant: precision 1/3 = 0.333, and its ceiling is R/k. The first hit is rank 3: reciprocal rank 1/3 = 0.333. Pass checklist: you divided recall by R and precision by k.

Medium. One answer: Express delivery takes 4 business days [kc-shipping-v1:c0]. shown with kc-shipping-v1:c0, whose text says express takes 2 business days and standard takes "4 to 7". The "4" is in the chunk, so the set-based number check passes, and so do the words.

from evidencedesk.retrieval.evidence import check_sentences, citation_scores

text = {c.id: c.text for c in cat.chunks}
claim = "Express delivery takes 4 business days [kc-shipping-v1:c0]."
print(citation_scores(check_sentences(claim, {"kc-shipping-v1:c0": text["kc-shipping-v1:c0"]})))

{'coverage': 1.0, 'resolve': 1.0, 'support': 1.0}

What catches it: a person reading the sentence next to the chunk, or a claim-level entailment check calibrated against human labels. Pass checklist: the three rates are all 1.0; you named a check that reads meaning, and you did not claim the Beginner check is wrong, only limited.

Hard. Six. Use the exact test on 21 questions where n questions improve and none get worse:

from evidencedesk.retrieval.metrics import paired

ref = {f"q{i}": 0.0 for i in range(21)}
for n in range(4, 8):
    print(n, "questions flip upward, none back: p =", paired(ref, {**ref, **{f"q{i}": 1.0 for i in range(n)}}).p)

4 questions flip upward, none back: p = 0.125
5 questions flip upward, none back: p = 0.062
6 questions flip upward, none back: p = 0.031
7 questions flip upward, none back: p = 0.016

Four one-sided flips give 0.125 and five 0.062, so six is the smallest number below 0.05: 29% of the answerable set, with none moving the other way. A change that helps 3 questions and hurts 1 can never be called significant here. For a release gate: hard rules for what must never break (critical cases), generous margins for averages, and a small movement is a request for review, not a verdict (Robust Metrics and Release Gates goes further). Pass checklist: you used the exact test; you saw that flips, not the size of each change, decide it; you tied it to the gate.

Summary

• A cross-encoder scores each (query, passage) pair together; it costs one run per pair, so it reorders a small candidate set, and its raw score is a ranking score, not a probability. The kit tests the interface with deterministic stand-ins; no real model ran here.
• Recall@k, precision@k, MRR and nDCG answer different questions. Name the relevance threshold and the nDCG variant (gain 2^grade - 1, discount log2(1 + rank); libraries differ), and leave undefined questions out instead of scoring them zero.
• With 24 labels, report an interval, the better/worse counts and the slice sizes. A one- or two-question gap is noise, and an observed 4 to 0 is still p = 0.125.
• Reranking can hurt: lost first-stage knowledge, truncated passages, extra cost. Size candidates and final context against the noise, and ablate one setting at a time.
• Dedupe by words and numbers, replace and rebuild on change, filter by tenant before ranking (with a control test), and check citations for coverage, resolution and support, knowing the word check misses negation.
• Feed a retrieval run into the Beginner release gate with critical cases and a noise-aware review line.

Next: MCP Architecture and Local Server. The chapter on retrieval is complete for now: the next page turns to the Model Context Protocol, where a host application, a client and a server expose tools to a model, and starts a local server you can test.