Chapter 1 · Advanced RAG: Hybrid Retrieval and Reranking
Hybrid Retrieval and Error Analysis
- Page 1 of 22
- 27 min read
The Beginner tutorial ended with an evaluation baseline that retrieved the right document for 84.6% of its answerable questions and failed four cases (Evaluation Datasets and Deterministic Tests). That says that something fails, not where. Nadia in Dhaka hears "the bot is wrong about returns" from her support team. Is the policy missing from the index, ranked fifth, dropped by a threshold, mangled by the model, or cited wrongly? Each has a different fix. This page builds the diagnosis layer (a stage-by-stage check and an error taxonomy), then compares keyword, vector and fused retrieval on the same hand-made labels.
The new code is the subpackage evidencedesk/retrieval/. It imports the Beginner modules (bm25, embeddings, index, chunking, citations) and rebuilds none. One warning before any number: "dense" here means the toy embedder from Embeddings and Similarity, hashed character trigrams that match letters, not meaning. The results show the method. They are not production numbers, and no real embedding model was run. With 21 answerable questions, one question moves a rate by 4.8 points.
What you will learn
- How to find which pipeline stage lost the evidence, and a five-label error taxonomy to count it.
- How to compare keyword and vector search on identical labels, and fuse them with reciprocal rank fusion (RRF) that you can check by hand.
- Why filters belong before ranking, and how to test that stale or restricted text never gets in.
- How chunking by structure treats tables and code, and how a two-stage pipeline is shaped.
Diagnose the baseline pipeline
A RAG answer passes five stages. A fact lost at one stage cannot be repaired by a later one, so check them in order and fix the earliest failure.
| Stage | Question |
|---|---|
| ingestion | Is the answer text whole inside some indexed chunk (Label.grade_of)? |
| retrieval | Is an answering chunk in the top k, and at which rank? |
| context | Did it survive the score floor, de-duplication and budget? |
| generation | Does the claim follow from the cited text (check_support)? |
| citation | Does each cited ID name a chunk the model was shown (check_citations)? |
The check needs ground truth, so the kit gained data/retrieval/labels.jsonl: 24 questions (the 16 of rag_cases.jsonl with the same ids, plus 8 new ones). Each label was written by reading the documents, before any search was run. A label names no chunk IDs, because a different chunker would break them. It holds a short evidence phrase copied from the document, a grade (2 answers the question, 1 is useful context), optional phrases the same chunk must also hold (a table row needs its header) and a reason. Label.grade_of(chunk) returns the best grade among the phrases that sit whole in the chunk.
First the Beginner baseline: vector search over paragraph chunks, current documents only, top 3, cosine at least 0.3. Three new documents (a torque and pressure chart, a troubleshooting guide, a staff-only note) make the corpus a little harder.
from evidencedesk.retrieval.catalog import Scope, load_catalog
from evidencedesk.retrieval.labels import load_labels
from evidencedesk.retrieval.report import evaluate
from evidencedesk.retrieval.searchers import DenseSearcher
labels = load_labels()
cat = load_catalog("paragraph") # the Beginner chunks, plus three new documents
dense = DenseSearcher(cat.chunks)
rows = evaluate(dense, cat, labels, Scope(), k=3, floor=0.3) # the Beginner baseline: current only, cosine >= 0.3
print(f"{len(rows)} questions, {sum(r.failed for r in rows)} fail")
for r in rows:
if r.failed:
f = r.findings[0]
print(f"{r.id:<5}{f.stage:<10}{f.label:<17}{f.detail}")24 questions, 5 fail
rc03 context missed_evidence retrieved at rank 1 but dropped before the prompt
rc06 retrieval missed_evidence answer not in the top 3; rank 1 is kc-privacy-v1:c0
rc08 retrieval missed_evidence answer not in the top 3; rank 1 is kc-sizing-v1:c1
rc15 context irrelevant_match 3 chunk(s) in the prompt; the corpus has no answer
h01 retrieval missed_evidence answer not in the top 3; rank 1 is kc-torque-v1:c0Five of 24 fail, for five different reasons. rc03 ("RD2-24") is not a retrieval failure: the recall notice is rank 1 and the score floor throws it away at 0.295 (the guard of RAG Reliability and Evaluation Data). In rc06 and rc08 ("money back", "voucher") vector search cannot connect the question to the refund or gift-card text. rc15 has no answer, yet three chunks pass the floor. h01 is a table row. (The Beginner suite also applied a margin rule and checked answers, so its failure list differs.)
The other stages need a broken run: a chunker with windows far too small (ingestion), a hand-written answer with a wrong number, and one citing a chunk that was never shown. The model is a stand-in; real models are less predictable, which is why you test.
from evidencedesk.retrieval.candidates import ranked
from evidencedesk.retrieval.report import run_trace
from evidencedesk.retrieval.taxonomy import Trace, diagnose
rc01 = next(x for x in labels if x.id == "rc01")
cut = load_catalog("fixed", size=60, overlap=0) # windows far too small: the evidence is cut in two
hit = ranked([(cat.by_id["kc-returns-v3:c0"], 0.55)], "dense")
cases = {
"ingestion": run_trace(rc01, DenseSearcher(cut.chunks), cut, Scope(), 3),
"generation": Trace(tuple(cat.chunks), hit, hit, "Return within 60 days.", ("kc-returns-v3:c0",)),
"citation": Trace(tuple(cat.chunks), hit, hit, "Return within 30 days of delivery.", ("kc-returns-v3:c1",)),
}
for name, trace in cases.items():
print(f"{name:<11}", [(f.stage, f.label) for f in diagnose(rc01, trace)])
lost = sum(not any(x.grade_of(c) == 2 for c in cut.chunks) for x in labels if x.answerable)
print(f"with windows of 60 characters, {lost} of 21 answers are whole in no chunk")ingestion [('ingestion', 'missed_evidence')]
generation [('generation', 'unsupported_answer')]
citation [('citation', 'unsupported_answer')]
with windows of 60 characters, 16 of 21 answers are whole in no chunkCreate a retrieval error taxonomy
A stage says where to fix; a label says what you saw. One finding is a (stage, label) pair. diagnose returns findings sorted by stage, so the first is the root cause.
| Label | Meaning | Detected by |
|---|---|---|
missed_evidence | The answer exists but did not reach the next stage | not whole in any chunk, not in the top k, or dropped before the prompt |
irrelevant_match | Something else outranks the answer, or fills the prompt when there is no answer | rank 1 has grade 0; any context for an unanswerable question |
stale_document | A superseded document is a candidate | status != "current" |
duplicate_chunks | Near-identical chunks share the prompt | word overlap of 0.8 or more (Beginner overlap) |
unsupported_answer | The claim is not backed by the cited text | missing numbers, low key-word share, or a citation not shown |
# file: evidencedesk/retrieval/taxonomy.py (excerpt)
STAGES = ("ingestion", "retrieval", "context", "generation", "citation")
LABELS = ("missed_evidence", "irrelevant_match", "stale_document", "duplicate_chunks", "unsupported_answer")
@dataclass(frozen=True)
class Finding:
stage: str # one of STAGES: where the fix belongs
label: str # one of LABELS: what the symptom is
detail: str
@dataclass(frozen=True)
class Trace:
"""What one run did, step by step. Built by the pipeline, read by diagnose()."""
indexed: tuple[Chunk, ...] # every chunk the index holds
retrieved: RankedList # what search returned
context: RankedList # what survived into the prompt (score floor, de-duplication, token budget ...)
answer: str | None = None # the model's claim, if a model was called
cited: tuple[str, ...] = () # chunk ids the model citedThe rest of taxonomy.py is in the kit. A valid-looking citation is not a pass: check_citations separates an ID that exists but was not shown from one that exists nowhere, and the claim must still fit the text. One label row, the stale-document trap:
{"id": "rc09", "question": "Are refunds paid as store credit?", "tags": ["stale"], "answerable": true,
"gold": [{"section": "kc-returns-v3#How refunds are paid", "evidence": "Refunds go back to the original payment method within 5 business days", "grade": 2}],
"reason": "The current policy says original payment method. The superseded v2 says store credit: retrieving it gives a confident wrong yes."}Counting labels over all 24 questions shows what a filter buys. Then one more failure is made on purpose: a re-import that kept two copies of one chunk.
from dataclasses import replace
from evidencedesk.retrieval.catalog import Catalog
from evidencedesk.retrieval.report import label_counts
open_ = label_counts(evaluate(dense, cat, labels, None, k=3, floor=0.3)) # no filter at all
shut = label_counts(evaluate(dense, cat, labels, Scope(), k=3, floor=0.3))
print(f"{'stage':<11}{'label':<17}{'no filter':>10}{'filtered':>10}")
for key in sorted(open_.keys() | shut.keys()):
print(f"{key[0]:<11}{key[1]:<17}{open_[key]:>10}{shut[key]:>10}")
twin = replace(cat.by_id["kc-returns-v3:c0"], id="kc-returns-v3:c0-copy") # a re-import that kept both copies
dup = Catalog(cat.chunks + [twin], cat.audience)
t = run_trace(rc01, DenseSearcher(dup.chunks), dup, Scope(), 3)
print([c.chunk.id for c in t.retrieved])
print([(f.stage, f.label) for f in diagnose(rc01, t)])stage label no filter filtered
context irrelevant_match 1 1
context missed_evidence 1 1
retrieval irrelevant_match 6 4
retrieval missed_evidence 3 3
retrieval stale_document 7 0
['kc-returns-v3:c0', 'kc-returns-v3:c0-copy', 'kc-warranty-v2:c0']
[('context', 'duplicate_chunks')]Unfiltered, seven questions have a superseded chunk in the top 3 (rc09 ranks the old store-credit rule first); with Scope() none do. The other labels remain: they need better retrieval, not a filter. The duplicate takes two of three prompt slots (Context Quality, Budgets, and Reproducibility removed such repeats with drop_duplicates).
Build a BM25 keyword retrieval baseline
This is the BM25 of RAG Reliability and Evaluation Data, unchanged. SparseSearcher only gives it the common interface and builds it with BM25.from_chunks, so it indexes exactly the text the vector index embeds. For one term the score is idf * tf / (tf + k1 * (1 - b + b * length / avg_length)), with k1 = 1.2 and b = 0.75; a rare word weighs more than a common one. The block checks it by hand on the exact identifier:
from evidencedesk.retrieval.searchers import SparseSearcher
sparse = SparseSearcher(cat.chunks)
bm = sparse.bm25
print(f"{bm.n} chunks, average length {bm.avg_len:.1f} words")
for term in ("rd2-24", "return", "bike"):
print(f"{term:<8}in {bm.df[term]:>2} chunks idf {bm.idf(term):.2f}")
for q in ("RD2-24", "How do I get my money back?"):
print(q)
for s in (sparse, dense):
print(f" {s.name:<7}", [(c.chunk.id.removeprefix("kc-"), round(c.score, 2)) for c in s.search(q, 3, cat.allowed(Scope()))])
n, hand = bm.length["kc-recalls-v1:c0"], 3.56 * 1 / (1 + 1.2 * (1 - 0.75 + 0.75 * bm.length["kc-recalls-v1:c0"] / bm.avg_len))
print(f"by hand for RD2-24: chunk of {n} words, 3.56 * 1 / (1 + 1.2 * (0.25 + 0.75 * {n} / {bm.avg_len:.1f})) = {hand:.2f}")52 chunks, average length 21.0 words
rd2-24 in 1 chunks idf 3.56
return in 2 chunks idf 3.05
bike in 12 chunks idf 1.44
RD2-24
sparse [('recalls-v1:c0', 1.5)]
dense [('recalls-v1:c0', 0.29), ('troubleshoot-v1:c1', 0.22), ('troubleshoot-v1:c0', 0.19)]
How do I get my money back?
sparse [('returns-v3:c2', 1.72)]
dense [('privacy-v1:c0', 0.22), ('shipping-v1:c1', 0.22), ('returns-v3-bn:c1', 0.18)]
by hand for RD2-24: chunk of 25 words, 3.56 * 1 / (1 + 1.2 * (0.25 + 0.75 * 25 / 21.0)) = 1.50"RD2-24" is in one chunk, so its idf is high and BM25 returns that chunk alone. A word absent from every chunk scores zero: BM25 is silent where vector search always returns its nearest neighbours. Caveats: no stemming ("returnable" misses "return"), and idf counts chunks a filter will hide. A three-document fixture with the hand calculation is in tests/test_retrieval_fusion.py.
Compare dense and sparse retrieval
A fair comparison changes one thing: same chunks, same embed text, same filter, same labels, same k. Hit@k means "an answering chunk is among the top k" (what the Beginner page called recall@k, at document level). The next page adds recall@k, precision@k, MRR and nDCG (the textbook variant: gain 2^rel - 1, discount log2(1 + rank); libraries differ).
from evidencedesk.retrieval.report import hit_at
TAGS = ["exact_term", "paraphrase", "bangla", "structured"]
by = {s.name: evaluate(s, cat, labels, Scope(), k=5) for s in (sparse, dense)}
print(f"{'tag':<12}{'n':>3}{'sparse@1':>10}{'sparse@3':>10}{'dense@1':>9}{'dense@3':>9}")
for tag in TAGS + [None]:
cells = [hit_at(by[s], k, tag)[0] for s in ("sparse", "dense") for k in (1, 3)]
print(f"{tag or 'all':<12}{hit_at(by['sparse'], 1, tag)[1]:>3}{cells[0]:>10}{cells[1]:>10}{cells[2]:>9}{cells[3]:>9}")
rank = {s: {r.id: r.rank for r in rows} for s, rows in by.items()}
for a, b in (("sparse", "dense"), ("dense", "sparse")):
only = [q for q in rank[a] if rank[a][q] and rank[a][q] <= 3 and not (rank[b][q] and rank[b][q] <= 3)]
print(f"top 3 has the answer for {a} but not {b}:", only)tag n sparse@1 sparse@3 dense@1 dense@3
exact_term 4 3 4 2 4
paraphrase 6 3 4 4 4
bangla 3 2 3 3 3
structured 6 4 6 3 5
all 21 15 19 14 18
top 3 has the answer for sparse but not dense: ['rc06', 'h01']
top 3 has the answer for dense but not sparse: ['rc05']Keyword search wins where words match exactly (rc06 matches "back" in "go back"; h01). Vector search wins rc05, "returnable after a month", but be honest about why: the toy embedder shares letter trigrams of "returnable" with "return". That is spelling, not meaning. Neither finds the voucher question (rc08); it needs a synonym that no method here knows. Dense search also finds "RD2-24" easily, which says something about trigrams, not about real models; they may treat identifiers differently, and we did not run one. The totals differ by one question (19 against 18 at hit@3): a tie.
Fuse ranked lists
Scores of two methods cannot be added: BM25 is unbounded, cosine lies between -1 and 1. Reciprocal rank fusion uses only ranks: score(d) = sum over lists of 1 / (k + rank of d in that list), rank starting at 1.
Tool note (as of 2026-10-12): RRF is from Cormack, Clarke and Buettcher, SIGIR 2009. They fixed
k = 60in a pilot, found it near-optimal but "not critical", and used k to limit the impact of high rankings by outlier systems. Their gains of about 4 to 5% over the best single system come from 2009 TREC and LETOR data and do not carry over to LLM-era hybrid search. Check the current documentation of any library you use for its own default.
# file: evidencedesk/retrieval/fusion.py (excerpt)
K = 60 # the paper's constant: fixed in a pilot, "near-optimal, but the choice was not critical"
def rrf(lists: Sequence[RankedList], k: int = K, top: int | None = None, source: str = "hybrid") -> RankedList:
"""score(d) = sum over the lists that contain d of 1 / (k + rank of d in that list). Scores are ignored.
Exact fractions make ties real ties; ties go to the better best-rank, then to the chunk id, so output is stable.
"""
if k < 0:
raise ValueError("k must not be negative")
total: dict[str, Fraction] = {}
first: dict[str, Candidate] = {}
parts: dict[str, list[Contribution]] = {}
for lst in lists:
for rank, cand in enumerate(lst, 1): # the position in the list, so a hand-built list works too
cid = cand.chunk.id
total[cid] = total.get(cid, Fraction(0)) + Fraction(1, k + rank)
first.setdefault(cid, cand)
parts.setdefault(cid, []).append(Contribution(cand.source, rank, cand.score))
best = {cid: min(p.rank for p in ps) for cid, ps in parts.items()}
order = sorted(total, key=lambda cid: (-total[cid], best[cid], cid))
fused = [Candidate(first[cid].chunk, float(total[cid]), n, source, tuple(parts[cid])) for n, cid in enumerate(order, 1)]
return fused[:top]Exact fractions make ties real ties, and the tie rule (best rank, then ID) keeps output stable. Each fused candidate keeps its sources' raw scores in parts, because RRF itself discards them. The hand-calculated cases are the fixture data/retrieval/rrf_examples.json. For lists [d1,d2,d3] and [d3,d1,d4]: d1 = 1/61 + 1/62 = 0.032522 and d3 = 1/63 + 1/61 = 0.032266, so d1 wins although d3 is first in one list. Two more cases show k: with k = 1 a first place in one list beats a fourth place in two lists; with k = 60 the document both lists contain wins.
import json
from fractions import Fraction
from evidencedesk.retrieval.candidates import ids, ranked
from evidencedesk.retrieval.fusion import rrf
from evidencedesk.retrieval.labels import DATA
class Doc: # the smallest stand-in for a chunk: fusion only reads .id
def __init__(self, id): self.id = id
for ex in json.loads((DATA / "rrf_examples.json").read_text(encoding="utf-8"))[:2]:
lists = [ranked([(Doc(i), 0.0) for i in names], "x") for names in ex["lists"]]
got = [(c.chunk.id, c.score) for c in rrf(lists, ex["k"])]
hand = [(i, float(sum(Fraction(p.strip()) for p in expr.split("+")))) for i, expr in ex["expected"]]
print(ex["id"], "k =", ex["k"], "same as by hand:", all(a[0] == b[0] and abs(a[1] - b[1]) < 1e-12 for a, b in zip(got, hand)))
print(" ", [(i, round(s, 5)) for i, s in got])two-lists k = 60 same as by hand: True
[('d1', 0.03252), ('d3', 0.03227), ('d2', 0.01613), ('d4', 0.01587)]
agreement-beats-one-first-place k = 60 same as by hand: True
[('q', 0.03226), ('p', 0.01639), ('r', 0.01639)]Now the real lists: an exact identifier, a paraphrase and the table question. Where does the answer sit in each list and in the fused list?
from evidencedesk.retrieval.searchers import HybridSearcher
hybrid = HybridSearcher(sparse, dense)
allowed = cat.allowed(Scope())
for q in ("RD2-24", "Is a bike returnable after a month?", "What torque should the stem bolts have on a size M?"):
lab = next(x for x in labels if x.question == q)
fused = hybrid.search(q, 20, allowed)
top = next(c for c in fused if lab.grade_of(c.chunk) == 2)
where = {p.source: p.rank for p in top.parts}
print(f"{q[:34]:<36}answer: sparse rank {where.get('sparse', '-')}, dense rank {where.get('dense', '-')}, hybrid rank {top.rank}")
top = lambda q: round(dense.search(q, 1, allowed)[0].score, 3)
print("best dense score: unanswerable 'electric scooters'", top("Do you sell electric scooters?"),
"| answerable 'RD2-24'", top("RD2-24"), "| 'money back'", top("How do I get my money back?"))RD2-24 answer: sparse rank 1, dense rank 1, hybrid rank 1
Is a bike returnable after a month answer: sparse rank 4, dense rank 1, hybrid rank 2
What torque should the stem bolts answer: sparse rank 3, dense rank 12, hybrid rank 5
best dense score: unanswerable 'electric scooters' 0.456 | answerable 'RD2-24' 0.295 | 'money back' 0.224Fusion rescues the paraphrase (4th by keyword, 1st by vector, 2nd fused) but is not free: the table row is 3rd and 12th in the two lists and 5th fused, because the weak vector list dilutes the good keyword list. The last line shows another cost. Without scores there is no "how sure": the best cosine for the unanswerable scooter question (0.456) is higher than for "RD2-24" (0.295) and "money back" (0.224), which have answers. A high similarity score is a signal, not proof, and no threshold on it separates these cases.
Apply metadata and access filters early
Two kinds of text must never enter a candidate set: superseded text, and text the asker may not see. The kit marks the second kind in the front matter (audience: staff; the Beginner loaders ignore the key). A Scope turns metadata into the set of allowed chunk IDs, and every searcher takes that set, so only allowed chunks are ranked:
# file: evidencedesk/retrieval/catalog.py (excerpt)
@dataclass(frozen=True)
class Scope:
"""What a request is allowed to see. The default is a public customer asking about current policy."""
statuses: frozenset[str] = frozenset({"current"})
audiences: frozenset[str] = frozenset({"public"})
langs: frozenset[str] | None = None # None = any language# file: evidencedesk/retrieval/searchers.py (excerpt)
class DenseSearcher:
name = "dense"
def __init__(self, chunks: list[Chunk], model: Model | None = None) -> None:
self.chunks, self.model = {c.id: c for c in chunks}, model or Model.toy()
self.index, _ = VectorIndex.build([c.to_passage() for c in chunks], self.model, lambda: "fixed-clock")
def search(self, query: str, k: int = 5, allowed: set[str] | None = None) -> RankedList:
"""Filter first, then rank only the allowed vectors. Zero or negative cosine is not a match."""
self.index.meta.check(self.model)
rows = [n for n, cid in enumerate(self.index.ids) if allowed is None or cid in allowed]
if unusable_reason(query) or not rows:
return []
hits = top_k(self.model.embed_batch([query])[0], self.index.matrix[rows], [self.index.ids[n] for n in rows], k)
return ranked([(self.chunks[h.passage_id], h.score) for h in hits if h.score > 0], self.name)The anti-pattern is to take the top k and drop what is not allowed afterwards (post_filter in the kit):
from evidencedesk.retrieval.searchers import post_filter
q = "Can I return a bike after 40 days?"
everyone = cat.allowed(Scope(audiences=frozenset({"public", "staff"}), statuses=frozenset({"current", "superseded"})))
short = lambda found: [c.chunk.id.removeprefix("kc-") for c in found]
print("no filter ", short(hybrid.search(q, 3, everyone)))
print("post-filter", short(post_filter(hybrid, q, 3, allowed)))
print("pre-filter ", short(hybrid.search(q, 3, allowed)))
leak = "goodwill refund store manager"
print(leak, "->", short(hybrid.search(leak, 3, everyone))[:2], "| post-filter:", short(post_filter(hybrid, leak, 3, allowed)))
print(len(everyone), "chunks in the index,", len(allowed), "visible to a public customer asking for current policy")no filter ['returns-v2:c0', 'returns-v3:c0', 'staff-goodwill-v1:c0']
post-filter ['returns-v3:c0']
pre-filter ['returns-v3:c0', 'returns-v3:c2', 'battery-v1:c1']
goodwill refund store manager -> ['staff-goodwill-v1:c0', 'staff-goodwill-v1:c1'] | post-filter: []
52 chunks in the index, 48 visible to a public customer asking for current policyPost-filtering returned one chunk instead of three, and for the leak query nothing, because the only hits were forbidden. Pre-filtering fills the list from what is allowed. test_retrieval_filters.py runs every labelled question plus two attack queries through all three searchers at four values of k and asserts that no superseded or staff chunk comes back. This is access control in code before ranking, not a prompt asking the model to ignore text (a prompt is not a guardrail). A vector database may filter inside an approximate index; read its documentation and test that a filtered query still returns k results. The next page extends the idea to tenants.
Choose chunking by document structure
Chunking decides what a hit is. Four chunkers, one set of labels, hybrid search: fixed (200-character windows over the whole body, blind to headings), section (one chunk per ## heading), paragraph (the Beginner chunker) and structure (paragraphs, plus tables and code kept as units, next section). "Whole" counts answerable questions for which some chunk holds the answer whole (an ingestion check); "context chars" is the average size of the top 3 chunks, a proxy for tokens and cost.
def context_chars(searcher, c): # characters the model would read: the top 3 chunks
sizes = [sum(len(x.chunk.text) for x in searcher.search(l.question, 3, c.allowed(Scope()))) for l in labels]
return sum(sizes) / len(sizes)
print(f"{'strategy':<10}{'chunks':>7}{'whole':>7}{'hit@1':>7}{'hit@3':>7}{'hit@3 tables/code/long':>24}{'context chars':>15}")
for name in ("fixed", "section", "paragraph", "structure"):
c = load_catalog(name)
h = HybridSearcher(SparseSearcher(c.chunks), DenseSearcher(c.chunks))
whole = sum(any(x.grade_of(ch) == 2 for ch in c.chunks) for x in labels if x.answerable)
rows = evaluate(h, c, labels, Scope(), k=5)
print(f"{name:<10}{len(c.chunks):>7}{whole:>5}/21{hit_at(rows, 1)[0]:>7}{hit_at(rows, 3)[0]:>7}"
f"{hit_at(rows, 3, 'structured')[0]:>16}/{hit_at(rows, 3, 'structured')[1]}{context_chars(h, c):>15.0f}")strategy chunks whole hit@1 hit@3 hit@3 tables/code/long context chars
fixed 46 20/21 16 19 5/6 562
section 44 21/21 18 20 6/6 517
paragraph 52 21/21 16 19 5/6 382
structure 54 21/21 18 19 5/6 386Only fixed windows lose an answer: they cut the header off the stem-bolt row, so the row no longer says which column is size M. Whole sections hit best, but cost 517 characters of context against 382 for paragraphs; this corpus is mostly one-paragraph sections, so the strategies barely differ, and a one-question gap is noise. Choose by your documents: short sections suit section chunks, long multi-topic sections suit paragraphs, tables and code need their own rule, and fixed windows are a fallback for unstructured text.
Handle tables, code, and structured documents
A table cell without its header is a number with no meaning, and a command with a missing line does not run. chunk_structured first splits each section into text, table (lines starting with |) and fenced-code blocks (split_blocks). Then text blocks are packed up to 200 characters; a code block is never cut and keeps its line breaks; a table is cut only between rows and every piece repeats the header. Each chunk records its source lines, so a citation can say "Bolt torque, lines 16-18" (StructChunk adds a kind to the Beginner Chunk).
s = load_catalog("structure")
for x in s.chunks:
if x.id in ("kc-torque-v1:c1", "kc-torque-v1:c8"):
print(f"{x.id.removeprefix('kc-torque-v1:'):<3}{x.kind:<6}{x.location.label:<24}", repr(x.text[:58]))
flat = next(x for x in cat.chunks if x.doc_id == "kc-torque-v1" and "Stem bolts" in x.text)
print("paragraph chunker:", repr(flat.text[:70]))c1 table Bolt torque, lines 16-18 '| Part | Size M | Size L | Note |\n|---|---|---|---|\n| Stem'
c8 code Check a serial number, lines 37-40 '```bash\nkestrel-recall check --serial RD2-24-00417\nkestrel'
paragraph chunker: '| Part | Size M | Size L | Note | |---|---|---|---| | Stem bolts | 5 |'The last line is the Beginner paragraph chunker on the same table: it joins lines with spaces, so rows and commands run together. The hit rates above cannot see that loss; the citation and what the model reads are worse. PDF and web pages need a converter first; keep its headings and rows (for example as Markdown).
Implement a two-stage retrieval pipeline
Stage one is cheap and wide: fetch many candidates. Stage two is slower and narrow: score only those and keep a few for the prompt. TwoStage wraps any searcher and any object with a rerank method, and counts the candidates the second stage scored (its cost, counted instead of timed so output is repeatable).
# file: evidencedesk/retrieval/pipeline.py (excerpt)
class Reranker(Protocol):
def rerank(self, query: str, found: RankedList) -> RankedList: ... # same candidates, new scores and order
@dataclass
class TwoStage:
first: Searcher
reranker: Reranker
candidates: int = 20 # how many the first stage fetches
final: int = 3 # how many reach the context
name: str = "two_stage"
scored: int = field(default=0, init=False) # candidates the second stage had to score: its cost, counted not timed
def search(self, query: str, k: int | None = None, allowed: set[str] | None = None) -> RankedList:
broad = self.first.search(query, self.candidates, allowed)
self.scored += len(broad)
return self.reranker.rerank(query, broad)[: k or self.final]OverlapReranker is a deterministic stand-in that scores the share of query words found in the chunk. It is not a cross-encoder; a model that reads query and passage together is the next page. Varying the candidate count shows the limit of the design:
from evidencedesk.retrieval.pipeline import OverlapReranker, TwoStage
base = evaluate(hybrid, cat, labels, Scope(), k=3)
print(f"first stage alone: hit@1 {hit_at(base, 1)[0]}, hit@3 {hit_at(base, 3)[0]} of 21")
print("candidates answer among candidates after rerank hit@1 hit@3 chunks scored per question")
for n in (3, 10, 30):
two = TwoStage(hybrid, OverlapReranker(), candidates=n, final=3)
rows = evaluate(two, cat, labels, Scope(), k=3)
pool = hit_at(evaluate(hybrid, cat, labels, Scope(), k=n), n)[0]
print(f"{n:>10}{pool:>20}/21{hit_at(rows, 1)[0]:>19}{hit_at(rows, 3)[0]:>7}{two.scored / len(labels):>16.1f}")first stage alone: hit@1 16, hit@3 19 of 21
candidates answer among candidates after rerank hit@1 hit@3 chunks scored per question
3 19/21 17 19 3.0
10 20/21 18 20 10.0
30 20/21 18 20 21.7The second stage can only re-order what the first fetched, so the answer must be among the candidates (19 of 21 with 3, 20 of 21 with 10). With 10 candidates the same hybrid search moves from 16 to 18 answers at rank 1, for 10 scored chunks per question instead of 3; 30 candidates add nothing here. The next page measures candidate count and final context size properly.
Exercise: compare three retrieval strategies
Keyword-only, dense-only and hybrid, on the same chunks, filter and labels. Run the block and the tests, then write your recommendation before reading mine.
from evidencedesk.retrieval.report import label_counts
print(f"{'strategy':<9}{'hit@1':>6}{'hit@3':>6}{'hit@5':>6}{'answer outranked':>18}{'no answer in 5':>16}{'junk for unanswerable':>23}")
for s in (sparse, dense, hybrid):
rows = evaluate(s, cat, labels, Scope(), k=5)
c = label_counts(rows)
junk = sum(r.returned > 0 for r in rows if not r.answerable)
print(f"{s.name:<9}{hit_at(rows, 1)[0]:>6}{hit_at(rows, 3)[0]:>6}{hit_at(rows, 5)[0]:>6}"
f"{c[('retrieval', 'irrelevant_match')]:>18}{c[('retrieval', 'missed_evidence')]:>16}{junk:>20}/3")
missed = {s.name: [r.id for r in evaluate(s, cat, labels, Scope(), k=5) if r.answerable and r.rank is None] for s in (sparse, dense, hybrid)}
print("no answer in the top 5:", missed)strategy hit@1 hit@3 hit@5 answer outranked no answer in 5 junk for unanswerable
sparse 15 19 20 5 1 1/3
dense 14 18 19 5 2 3/3
hybrid 16 19 20 4 1 3/3
no answer in the top 5: {'sparse': ['rc08'], 'dense': ['rc08', 'h01'], 'hybrid': ['rc08']}python -m pytest tests/test_retrieval_fusion.py tests/test_retrieval_labels.py tests/test_retrieval_taxonomy.py \
tests/test_retrieval_filters.py tests/test_retrieval_structure.py tests/test_retrieval_pipeline.py 2>&1 | tail -1 | sed 's/ in [0-9.]*s//'40 passedMy recommendation: use hybrid with a pre-filter as the default. It has the best hit@1 (16 against 15 and 14) and matches the better single method at hit@3 and hit@5. In the top 5 it misses only the voucher question, as keyword search does (dense also misses the table row). But it is not a clear win: the strategies differ by one or two questions out of 21; hybrid still ranks the table row fifth; and for unanswerable questions hybrid and dense return chunks every time (3 of 3). Returning nothing needs a separate sufficiency check, which no ranking method provides. Do not tune k or the depth on these 21 questions and then report them as your result. After that: a real embedding model (not run here), a reranker, more labelled questions.
The kit gained evidencedesk/retrieval/ (nine small modules), data/retrieval/ and tests/test_retrieval_*.py.
Where this shows up at work
A support team reports "the bot is wrong". The engineer on call opens the trace of one failing question, finds the stage, and files the bug with the right owner: ingestion for a broken PDF converter, retrieval for a vocabulary gap, context for a threshold. Counting labels over fifty failures a week shows the pattern and turns "improve RAG" into tickets. The labelled set then guards every change, like the suite of Rubrics, Regression, and Release Gates.
Common mistakes
- Tuning on the test set. Choosing
k, depth or a threshold on the 21 questions and quoting their score. Hold some out. - Filtering after ranking. It gives short or empty results and lets forbidden text into logs and later stages.
- Reading an RRF score as confidence. It depends only on ranks. Keep the raw scores (
parts) if you need a gate, and know they separate answerable questions badly. - Labels tied to chunk IDs or whole documents. Chunk IDs change with the chunker; document-level labels hide a cut table. Label evidence text.
- Calling toy results real. A trigram embedder is not an embedding model. Say what was run.
Try it yourself
- Easy (revisits Embeddings and Similarity): search for the mistyped identifier
"RD2-25"withSparseSearcherandDenseSearcher(allowed ids fromScope(), k = 3). Predict which one finds the recall notice, then run it. - Medium (revisits RAG Reliability and Evaluation Data): copy
labels.jsonlto a temporary file, add the question "Can you deliver to a post-office box?" with evidence phrase and reason, load it withload_labels(path), and run the three strategies on 25 questions. - Hard (revisits Local Indexes and Retrieval Quality): find the single dense-score threshold that best separates the 21 answerable from the 3 unanswerable questions (answer if the best score is at least t). How many of 24 decisions are wrong, and what does it tell you?
Answers
Easy. Keyword search returns an empty list: the token "rd2-25" is in no chunk. Vector search returns three chunks and ranks the recall notice second (cosine 0.24), behind the torque chart's code block, which contains "RD2-24-00417". Pass checklist: you predicted the empty keyword result; you saw that vector search copes with the typo only through shared letters and shows no sign that it is unsure.
Medium. The evidence phrase is "We cannot deliver to post-office boxes" in kc-shipping-v1#Where we deliver, grade 2, tag direct. Pass checklist: the phrase is copied exactly; load_labels accepts the file; the original labels.jsonl is untouched (a test pins its 24 rows).
Hard. Take the best dense score of each question under Scope() and try every threshold from 0.20 to 0.79:
top = {x.id: (dense.search(x.question, 1, allowed) or [None])[0] for x in labels}
best = min(((sum((c is not None and c.score >= t) != x.answerable for x in labels for c in [top[x.id]]), t)
for t in [n / 100 for n in range(20, 80)]))
print(f"best single dense threshold: {best[1]:.2f} -> {best[0]} of {len(labels)} wrong answer/abstain decisions")best single dense threshold: 0.28 -> 2 of 24 wrong answer/abstain decisionsTwo decisions stay wrong even at the best threshold: the scooter question scores above it, "money back" below it. And t was chosen on the same data, so 2 of 24 is optimistic. Pass checklist: you applied the scope first; you used one threshold for all questions; you concluded that a score floor alone cannot decide when to abstain, and that a check of whether the evidence supports the claim is needed.
Summary
- Diagnose in stage order (ingestion, retrieval, context, generation, citation), fix the earliest failing stage, and label each finding with one of five error types.
- Label relevance by evidence text with grades, so the same labels judge any chunker, retriever or fusion.
- Keyword and vector search fail on different questions; RRF merges them using ranks only (k = 60 from the 2009 paper), can be checked by hand, and is not free.
- Filter before ranking; keep structure (table headers, whole code blocks, source lines) when chunking; a second stage can only re-order the first stage's candidates.
- A toy embedder and 24 questions show the method, not production quality.
Next: Reranking and Retrieval Evaluation adds a cross-encoder style reranker, recall@k, precision@k, MRR and nDCG on graded labels, candidate and context sizing, de-duplication, freshness and tenant boundaries.