Chapter 9 · Advanced Context and Retrieval Operations
Retrieval Operations and Data Freshness
- Page 18 of 22
- 26 min read
On Monday Kestrel Cycles changes its returns window from 14 to 21 days. By Wednesday the English page says 21, the Bangla page still says 14, a cached answer says 14, and a customer holds a citation that points at text which is no longer there. Nothing crashed. The index was right on the day it was built, and the world moved. (An illustrative case.)
The previous page built the governed pipeline: Ingestion and Permission-Aware Retrieval. This page runs it over time, in the same subpackage evidencedesk/ingest/. Honest limits: the embedder is still the toy of Embeddings and Similarity, the question sets are small (every rate is a count out of a few cases, not a promise), the clock is fake and the latency is simulated.
What you will learn
- How to detect stale content and contradictory sources, and how to resolve a conflict by authority and recency without merging claims.
- How to test long-document and multilingual retrieval, and how to design a fallback that answers, broadens, asks or abstains.
- How to re-index on a schedule, change the embedding model or chunker behind a shadow index, and keep citations and caches correct across revisions.
- How to audit index health, recover from a failed run, mine failed queries into a labelled backlog, and gate every re-index with a regression suite.
- How to write the runbook and replay a content change end to end.
Detect stale content
Freshness is a policy, not a number. Four signals, each a reason to look and none a proof. Age: days since the text last changed, against a limit per origin (a permission change does not reset it, and an old page can be right). Source version: hash the listing and compare it with the active manifests without ingesting: changed, new, vanished or unreadable. Held back: the newer file sits in quarantine, so an older version still serves. Stale sibling: another page of the same tenant states a different value than the page that just changed. The last one is the Kestrel trap: the page that changed is fine, and the page nobody remembered is wrong.
from collections import Counter
from evidencedesk.ingest import fresh
from evidencedesk.ingest.lab import fresh as build, snapshot
kb, _ = build("v1") # the live index on day 0; the clock is fake and moves only when told
print("reconcile:", fresh.behind_source(kb, snapshot("v2")))
kb.clock.advance(6 * fresh.DAY)
kb.ingest(snapshot("v2"), prune=True) # day 6: the returns window changed from 14 to 21 days
kb.clock.advance(44 * fresh.DAY) # day 50
rows = fresh.stale_report(kb, kb.clock(), snapshot("v2"))
print(dict(sorted(Counter(s.code for s in rows).items())))
for s in rows:
if s.code == "stale_sibling" or s.source_id == "kc-shipping":
print(f"{s.source_id:<14}{s.code:<14}{s.detail}")reconcile: {'kc-recall-notice': 'new', 'kc-returns': 'changed', 'kc-stores': 'vanished'}
{'review_due': 9, 'stale_sibling': 1}
kc-returns-bn stale_sibling return_window_days=14 but kc-returns=21
kc-shipping review_due content unchanged for 50 days, limit 14 for website-crawlThe reconcile saw the change before any embedding, so it is cheap enough to run nightly. At day 50 nine sources are past their limit (a review queue, not a defect) and the Bangla page is flagged as a stale sibling.
Validate source quality
The contract of the previous page is the gate, and operations reuses its codes. Empty and malformed files stop at clean or parse (empty, bad_json, ragged_table); hidden text, secret- and personal-data shapes and mislabels stop at clean and classify; identical chunks are indexed once per tenant and ACL. What arrives later is quality no single file shows: duplicates that disagree (next section), age (above) and untrusted origin. Treat the origin as a trust level (the authority table below); a move to a less trusted origin is a review item.
Handle contradictory evidence
Two pages can be well formed and still disagree. claims.extract reads three numeric facts with rules and turns Bangla digits into numbers, so a Bangla page can contradict an English one. conflicts groups claims by tenant and subject: Harbor's 30 days is another company, not a contradiction. resolve prefers the higher authority of the origin, then the newer content (when the current text first appeared, not when the file was last touched), and otherwise leaves the conflict open: show every value with its source and version, and never average or merge.
from evidencedesk.ingest import conflict
for c in conflict.conflicts(kb):
print(conflict.resolve(kb, c).text())
same_day, _ = build("v1")
same_day.ingest(snapshot("v2"), prune=True) # both pages changed on the same fake day
print(conflict.resolve(same_day, conflict.conflicts(same_day)[0]).text())
print(conflict.AUTHORITY)kestrel/return_window_days: kc-returns=21, kc-returns-bn=14 -> prefer kc-returns (by recency), flag kc-returns-bn
kestrel/return_window_days: kc-returns=21, kc-returns-bn=14 -> UNRESOLVED, show all with provenance
{'help-centre-export': 3, 'spreadsheet-export': 3, 'store-api': 3, 'pdf-text-extract': 2, 'website-crawl': 2, 'partner-feed': 2, 'default': 1}On day 50 recency prefers kc-returns and flags the Bangla page. When both changed on the same day nothing can be decided, and the answer must say so. A resolution is advice for the page owner: the loser stays in the index until someone fixes it.
Test long-document retrieval
A fact buried in a long handbook can be lost in two different ways: chunking cuts it in two (no chunk holds the sentence), or its chunk exists but loses the top-k to look-alikes. The test separates them. longdoc.make_handbook() generates 14 sections of look-alike text (every section mentions bearings and service), buries four facts, and places one across a fixed-window edge, so no long file sits in the repository.
from evidencedesk.ingest import longdoc
doc = longdoc.make_handbook()
print(f"{'chunker':<14}{'chunks':>6}{'chars':>6} fact in section 0, 6, 9, 13: k=1 | k=3")
for name in longdoc.CHUNKERS:
n, avg, one = longdoc.survival(doc, name, 1)
three = longdoc.survival(doc, name, 3)[2]
print(f"{name:<14}{n:>6}{avg:>6} " + " ".join(f"{r.status:<7}" for r in one) + "| " + " ".join(f"{r.status:<7}" for r in three))chunker chunks chars fact in section 0, 6, 9, 13: k=1 | k=3
fixed 120/0 73 107 ok@1 split ok@1 ok@1 | ok@1 split ok@1 ok@1
fixed 120/40 101 112 ok@1 split ok@1 ok@1 | ok@1 split ok@1 ok@1
sentences 200 57 137 ok@1 missed ok@1 ok@1 | ok@1 ok@3 ok@1 ok@1
whole section 14 561 ok@1 ok@1 ok@1 ok@1 | ok@1 ok@1 ok@1 ok@1 Fixed windows split the section-6 fact whatever the overlap (40 characters is shorter than the sentence). Sentence packing keeps it whole, but at k = 1 look-alikes win and the fact appears only at rank 3. Whole sections never lose a fact and cost about four times the context (561 characters against 137). Survival is an input to the choice of chunker and k, not a verdict: re-run it whenever either changes.
Evaluate multilingual retrieval
The language of the question and the language of the evidence are separate axes. Twelve labelled cases are sliced by question script to evidence script (script_of counts letters: a hint, not language identification).
from evidencedesk.ingest import lang, regress
from evidencedesk.ingest.contract import DATA
v3, _ = build("v3") # the third snapshot (the Bangla page revised); the next section reuses it
cases = regress.load_cases(DATA / "lang_cases.jsonl")
for k, langs in ((1, ()), (3, ()), (3, ("bn",))):
print(f"k={k} lang filter={str(langs or '-'):<7}", {s: f"{c.k}/{c.n}" for s, c in lang.by_slice(v3, cases, k, langs).items()})k=1 lang filter=- {'bn->bn': '4/4', 'en->bn': '0/2', 'en->en': '1/3', 'mixed->bn': '1/3'}
k=3 lang filter=- {'bn->bn': '4/4', 'en->bn': '0/2', 'en->en': '2/3', 'mixed->bn': '3/3'}
k=3 lang filter=('bn',) {'bn->bn': '4/4', 'en->bn': '0/2', 'en->en': '0/3', 'mixed->bn': '3/3'}Bangla to Bangla works (4/4). English to Bangla fails (0/2): the pages share no words, and the toy embedder matches letters, not meaning. Read k = 1 first: the Bangla partition has five chunks, so mixed queries at k = 3 look perfect by near-guessing. The language filter narrows only: with bn the English questions drop to 0/3, so a wrong guess about the user's language removes the right answer. Use script as a preference, not a gate.
Provider note (as of 2026-10-12): the model card of
paraphrase-multilingual-MiniLM-L12-v2lists many languages but not Bengali. The card ofmultilingual-e5-small(MIT licence, 384 dimensions) listsbn, expectsquery:andpassage:prefixes, and warns that low-resource languages may do worse. Both download weights, so nothing here runs them. Check the current documentation and re-run these slices on your own data.
Design a retrieval fallback policy
When retrieval is weak, the system has four honest moves. Answer (with a caveat if the visible results contain an open conflict). Broaden: if a user filter produced nothing, drop the user's filter, never the security filter. Clarify: a one-word query gets the sections it might mean. Abstain: one message, No matching documents., for every kind of empty. The fused score cannot drive abstention: it is rank-based, and a top hit in both lists scores 2/61 = 0.0328 (the constant of 60 comes from the reciprocal-rank-fusion paper, see Hybrid Retrieval and Error Analysis) whatever it says. decide uses coverage, the share of the question's content words found in the best chunk.
from evidencedesk.ingest import fallback, regress
from evidencedesk.ingest.access import Filters
from evidencedesk.ingest.lab import CUSTOMER, HARBOR, STAFF
v3, _ = build("v3")
print("calibration (lowest answerable, highest unanswerable, n, n):", fallback.calibrate(v3, regress.load_cases()))
asks = [(v3, "brake lever recall", None), (v3, "returns", None), (v3, "Do you sell electric scooters?", None),
(kb, "ফেরত return window", None), (v3, "return a bike", Filters(kinds=("code",)))]
for base, q, f in asks:
d = fallback.decide(base, CUSTOMER, q, f)
print(f"{q:<32}{d.action:<8}{d.reason:<14}cover {d.coverage} {list(d.options)}")calibration (lowest answerable, highest unanswerable, n, n): (0.2, 0.0, 16, 3)
brake lever recall answer ok cover 1.0 []
returns clarify one_word cover 1.0 ['Exceptions', 'Who pays for the return label?', 'Return window']
Do you sell electric scooters? abstain low_coverage cover 0.0 []
ফেরত return window caveat open_conflict cover 0.333 []
return a bike answer broadened cover 1.0 []The calibration is honest about its size: the lowest answerable question scores 0.2, the highest unanswerable one 0.0, so the threshold 0.15 sits in a gap measured on 16 and 3 cases. A clean gap on three cases is luck, so treat the threshold as a hypothesis, and re-check it when the corpus or the embedder changes.
Schedule re-indexing and count its cost
Two triggers: change events, debounced so a burst of edits makes one run, and a periodic reconcile that catches what no event announced. Cost is counted first (texts embedded, partitions and chunks rebuilt) and priced afterwards from a hypothetical table of 0.002 credits per embedded text and 0.0001 per rebuilt chunk. Latency uses a simulated model, 0.005 s plus 0.0002 s per candidate chunk, which still teaches the shape: a bigger tenant partition costs more per query, and the security pre-filter keeps the candidates to what the caller may read.
from evidencedesk.ingest import ops
from evidencedesk.ingest.kb import KnowledgeBase
print(ops.plan_runs([0, 60, 120, 3000, 3030, 9000], 600, 86_400, 100_000))
scenarios = {"same listing again": snapshot("v1"), "one edit, one ACL change, one add, one delete": snapshot("v2")}
live, _ = build("v1")
for name, entries in scenarios.items():
r = ops.run_cost(live, entries)
print(f"{name:<46}embedded {r.embedded:>2} rebuilt {len(r.rebuilt)} partitions, {r.chunks_rebuilt:>2} chunks {r.credits} credits")
r = ops.run_cost(KnowledgeBase(), snapshot("v2"))
print(f"{'full rebuild':<46}embedded {r.embedded:>2} rebuilt {len(r.rebuilt)} partitions, {r.chunks_rebuilt:>2} chunks {r.credits} credits")[(720, 'event'), (3630, 'event'), (9600, 'event'), (86400, 'reconcile')]
same listing again embedded 0 rebuilt 0 partitions, 0 chunks 0.0 credits
one edit, one ACL change, one add, one delete embedded 2 rebuilt 1 partitions, 16 chunks 0.0056 credits
full rebuild embedded 18 rebuilt 2 partitions, 18 chunks 0.0378 creditsThe incremental run embeds 2 texts and rebuilds one partition (Harbor's is untouched); the full rebuild embeds all 18. Both rebuild BM25 statistics for the changed tenant, as the previous page said. Re-embedding everything is the price of a model change, in the next section.
Change the embedding model or the chunker
Vectors from two models never share an index, and a chunker change moves ids. The safe path is a shadow index: build a candidate from the full listing, send each live query to both (the user sees only live), compare on the fixed question set, cut over, and keep the old index for rollback. compare is also the answer to measuring retrieval drift: the same fixed questions before and after, with the paired sign test of Reranking and Retrieval Evaluation.
from evidencedesk.ingest.rewrite import load_cases
from evidencedesk.ingest.shadow import NEW_MODEL, Serving, build_candidate, compare
live, _ = build("v1")
fixed = load_cases() # the 15 fixed questions of the previous page
for name, options in (("new embedding model", dict(model=NEW_MODEL)), ("smaller chunks (80)", dict(max_chars=80))):
cand = build_candidate(snapshot("v1"), **options)
r = compare(live, cand, fixed)
print(f"{name:<21}hit {r.live_hit} -> {r.shadow_hit} better {len(r.delta.better)} worse {len(r.delta.worse)} same first source {r.same_top1} moved {r.moved_ids} embedded {r.embedded}")
serving = Serving(live)
serving.shadow = cand
for q in ("How long do I have to return a bike?", "returns", "express shipping"):
serving.read(CUSTOMER, q)
print("shadow reads:", serving.reads, "| disagreements:", serving.disagreements)
serving.cutover()
print("live is the candidate:", serving.live is cand, "| old index kept:", serving.previous is live)
serving.rollback()
print("rolled back:", serving.live is live)new embedding model hit 0.6 -> 0.533 better 0 worse 1 same first source 0.933 moved 0 embedded 18
smaller chunks (80) hit 0.6 -> 0.6 better 0 worse 0 same first source 1.0 moved 5 embedded 23
shadow reads: 3 | disagreements: 0
live is the candidate: True | old index kept: True
rolled back: TrueThe new model changes the first source for one question in 15 and loses one (0.6 to 0.533); with 15 questions that is a flip, not a trend. The smaller chunks move five chunk texts and change no answer, which is exactly why citations (below) need more than source:cN. During the window you pay twice: double storage and a second search per query.
Provider note (as of 2026-10-12): Google's embeddings documentation states that the embedding spaces of its two models,
gemini-embedding-001andgemini-embedding-2, are incompatible, so upgrading means re-embedding all data. The OpenAI embeddings page we read has no such sentence; treat "one model per index" as a rule of method, not as a vendor statement. Model names and limits change often. Check the current documentation.
Test citation stability across revisions
A chunk id such as kc-returns:c0 only means "the first chunk now". A Cite stores the source, the chunk id, the section anchor (authors keep headings stable), the version and the hash of the quoted text. resolve follows it to what is served today, rechecking permission:
# excerpt of kit/evidencedesk/ingest/stableid.py
def resolve(kb: KnowledgeBase, who: Principal, c: Cite) -> Resolved:
"""Follow an old citation to what is served today. Permissions are rechecked; a removed passage and a forbidden one look the same."""
m = kb.manifests.get(c.source_id)
if m is None or m.status != "active" or m.acl is None or not m.acl.permits(who):
return Resolved("gone", None, GONE)
mine = {cid: r for cid, r in kb.records.items() if r.source_id == c.source_id and r.acl.permits(who)}
for cid, r in sorted(mine.items()): # 1. the same text, wherever it sits now
if r.chunk_hash == c.chunk_hash:
return Resolved("current" if cid == c.chunk_id else "moved", hit_of(kb, cid), f"same text, now {cid} (v{r.version})")
for cid, r in sorted(mine.items()): # 2. the same section with other text: show it and say so
if slug(r.chunk.location.heading) == c.anchor:
return Resolved("revised", hit_of(kb, cid), f"the section '{c.anchor}' changed between v{c.version} and v{r.version}")
return Resolved("gone", None, GONE)Four outcomes: current (same text, same id), moved (same text, other id: the redirect), revised (same section, new text: show the new text and say it changed) and gone. A removed passage and a forbidden one give the same message.
from evidencedesk.ingest import stableid
base, _ = build("v1")
old = {cid: stableid.cite(base, cid) for cid in ("kc-returns:c0", "kc-returns:c1", "kc-stores:c0", "kc-goodwill:c0", "kc-shipping:c2")}
print(old["kc-returns:c0"])
base.ingest(snapshot("v2"), prune=True) # event 1: a content change
small = build_candidate(snapshot("v1"), max_chars=80) # event 2: another chunker, the same text
print(f"{'cited chunk':<16}{'after the content change':<26}after the chunker change")
for cid, c in old.items():
a, b = stableid.resolve(base, STAFF, c), stableid.resolve(small, STAFF, c)
print(f"{cid:<16}{a.status:<26}{b.status}" + (f" -> {b.hit.chunk_id}" if b.status == "moved" else ""))Cite(source_id='kc-returns', chunk_id='kc-returns:c0', anchor='return-window', version=1, chunk_hash='52da36fd')
cited chunk after the content change after the chunker change
kc-returns:c0 revised current
kc-returns:c1 current current
kc-stores:c0 gone revised
kc-goodwill:c0 gone current
kc-shipping:c2 current moved -> kc-shipping:c3The returns citation is revised (14 became 21), the store and goodwill citations are gone (deleted, and narrowed to team leads), and the shipping citation moved to c3 when the chunker changed. The weak spot is visible too: the stores table, cut differently, reads as revised. Character offsets in the citation would be the stronger anchor.
Cache retrieval results, and invalidate on change
CachedSearch (the cache of Cost and Latency Fundamentals, applied on the previous page) puts the epoch in the key, so every change drops every entry: safe and cold. TagCache is finer: entries are tagged with the sources they quoted and sync drops only those. Test it with the one change a tag cannot see, a new document that should now rank first. The honest metric is not the hit rate but stale hits: cached answers that differ from a fresh search.
from evidencedesk.ingest.access import CachedSearch
from evidencedesk.ingest.cachefine import TagCache, stale_hits
kc, _ = build("v1")
asks = [(CUSTOMER, q) for q in ("How long do I have to return a bike?", "Is my bike affected by RD2-24?", "brake lever recall", "return label")]
epoch, tag = CachedSearch(kc), TagCache(kc)
for cache in (epoch, tag):
for who, q in asks:
cache.ask(who, q)
report = kc.ingest(snapshot("v1") + [e for e in snapshot("v2") if e["source_id"] == "kc-recall-notice"]) # only a NEW document
print("tag cache dropped", tag.sync(report), "entries | outcomes", report.counts())
print("stale answers / answers served from the cache: epoch cache", stale_hits(epoch, kc, asks), "| tag cache", stale_hits(tag, kc, asks))tag cache dropped 0 entries | outcomes {'added': 1, 'quarantined': 1, 'unchanged': 11}
stale answers / answers served from the cache: epoch cache (0, 0) | tag cache (3, 4)Nothing was dropped, because no cached answer quoted the new notice, and three of the four cached answers are now stale. Prefer the coarse cache unless you can show the fine one serves no stale answers, and keep a short TTL either way.
Audit index health
audit runs after every ingestion and every night. It reuses verify from the previous page and adds late denials (should be 0), embedding-cache size against need, pending changes and the freshness signals. The strongest check is the consistency audit: build the index from scratch and compare it with the incremental one. Any difference means the incremental path drifted.
for name, state, detail in ops.audit(kb, kb.clock(), snapshot("v2")):
print(f"{state:<5}{name:<16}{detail}")
print("incremental vs full rebuild:", ops.incremental_vs_full(kb, snapshot("v2")))ok verify no findings
ok late denials 0
ok embedding cache 18 vectors, 18 needed
ok pending changes none
WARN review_due hb-recall, hb-returns, kc-goodwill, kc-payroll-note, kc-recall-notice, kc-returns, kc-returns-bn, kc-returns-faq, kc-shipping
WARN stale_sibling kc-returns-bn
ok conflict_open none
ok held_back none
incremental vs full rebuild: []Design recovery for failed ingestion
A run that dies between "records updated" and "index rebuilt" leaves a quiet half-state: search still works, and ranks against old text. Because the job is idempotent, recovery is: detect the damage, run the same listing again, verify. Better, stage the run on a copy so readers never see the half-state. The checkpoint is the manifest store itself, which is why a repeatable job needs no extra bookkeeping.
from evidencedesk.ingest import recover
from evidencedesk.ingest.verify import verify
r1, _ = build("v1")
r1.log = recover.crashing_log(3) # the run is killed after three sources
try:
r1.ingest(snapshot("v3"), prune=True)
except recover.Crash as error:
print("crash:", error)
r1.log = None
print("after the crash:", [f.code for f in verify(r1)], "| chunks that differ from a full rebuild:", len(ops.incremental_vs_full(r1, snapshot("v3"))))
before, after = recover.heal(r1, snapshot("v3"))
print("after one more run of the same listing:", [f.code for f in after])
safe, _ = build("v1")
try:
recover.staged(safe, snapshot("v3"), crash_after=2)
except recover.Crash:
print("staged run crashed; the live index is untouched:", verify(safe), "epoch", safe.epoch)crash: killed after 3 sources
after the crash: ['index_drift'] | chunks that differ from a full rebuild: 7
after one more run of the same listing: []
staged run crashed; the live index is untouched: [] epoch 1The crash left index_drift and seven chunks differing from a full rebuild; one more run healed it. The staged run crashed with the live index untouched. A real crash is rougher than this simulated one, which is why the audit exists.
Read the query logs, then mine the failures
Events carry the action, the reason, counts and a pseudonym, never the query. Only failed queries (abstain, clarify) are stored, redacted, in a separate short-lived store; analytics reuse Obs and summarise from Monitoring, Budgets, and Incidents, including its rule that a rate below the minimum sample is None. mine clusters failures by shared words and asks the operator view whether any document of the tenant covers them: covered (look at ranking, wording or access) or content_gap. A backlog row becomes a labelled case (retrieval.labels.Label, see Hybrid Retrieval and Error Analysis) only when a person decides the right answer.
from evidencedesk.ingest import mining
from evidencedesk.observe.events import EventLog
from evidencedesk.observe.privacy import ContentStore
from evidencedesk.usage import ManualClock
log, store = EventLog(ManualClock(1000.0)), ContentStore()
prod, _ = build("v1", log=log)
prod.ingest(snapshot("v2"), prune=True)
n = mining.replay(mining.LoggedSearch(prod, log, store), mining.load_traffic()) # 100 requests from a traffic file
total, by_tenant = mining.analytics(log)
print(f"{n} requests | refusal rate {total.refusal_rate} | simulated latency p50 {total.p50_s} s | harbor: {by_tenant['harbor'].n} requests, rate {by_tenant['harbor'].refusal_rate}")
items = mining.mine(log, store, prod)
for it in items[:5]:
print(f"{it.count:>2} asks, {len(it.members)} wording(s) {it.kind:<21}{it.question}")
by_q = {i.question: i for i in items}
print(mining.to_label(by_q["Do you sell electric scooters?"], 1).model_dump_json()[:150])
try:
mining.to_label(by_q["returns"], 2)
except ValueError as error:
print("not a label yet:", error.errors()[0]["msg"])100 requests | refusal rate 0.34 | simulated latency p50 0.007 s | harbor: 4 requests, rate None
8 asks, 2 wording(s) content_gap Do you deliver to Nairobi?
8 asks, 1 wording(s) covered returns
6 asks, 2 wording(s) content_gap Do you sell electric scooters?
5 asks, 2 wording(s) content_gap Is there a student discount?
3 asks, 1 wording(s) content_gap এক্সপ্রেস ডেলিভারি কত দিনে হয়?
{"id":"m01","question":"Do you sell electric scooters?","tags":["unanswerable"],"answerable":false,"gold":[],"reason":"mined from 6 failed asks by 1 u
not a label yet: Value error, an answerable question needs a grade-2 gold item, and only thenA content gap labels itself as unanswerable and goes straight into the regression set; a covered question raises until a person supplies the gold evidence. The top rows are a delivery area, a product line and a discount code that no page answers; Harbor's four requests give no rate at all, because the sample is below the minimum.
Gate every re-index with a regression suite
regress runs 22 fixed cases after every re-index: evidence in the top 3, no stale value (the number the change replaced), no leak, correct abstention. The decision comes from the release gate of Robust Metrics and Release Gates with a policy file, floors, a no-regress rule against the live baseline and three critical cases. The baseline is the live index judged against the new expectations, so it fails the changed-fact cases, as it should.
base = regress.run(live) # the live index (snapshot v1) is the baseline
print("baseline:", {m: f"{c.k}/{c.n}" for m, c in base.counts.items()})
for name in ("v2", "v3"):
candidate, _ = build(name)
d = regress.gate(regress.run(candidate), base)
print(f"{name}: {d.verdict.upper()} (exit {d.exit_code()})")
for r in d.reasons:
if r.level != "note":
print(f" {r.level:<7}{r.rule}: {r.text}")baseline: {'evidence_at_3': '8/16', 'stale_free': '0/3', 'no_leak': '2/4', 'abstain_correct': '3/3'}
v2: BLOCK (exit 1)
block critical: case r16 failed
block floor: evidence_at_3 0.812 (13/16) is below the floor 0.9
block floor: stale_free 0.667 (2/3) is below the floor 1.0
review flips: 1 cases passed in the baseline and fail now: q07
v3: REVIEW (exit 0)
review flips: 1 cases passed in the baseline and fail now: q07Snapshot v2 is blocked: the stale Bangla page still serves 14 days (critical case r16, and the stale-value check at 2/3), and the evidence floor fails at 13/16. Snapshot v3, which revises that page, goes to review: one paraphrase question flipped because a new document took the third slot from the right chunk. A person reads the flip, accepts or fixes it, and files it in the backlog. The same gate pattern runs on the corpus benchmark of Reranking and Retrieval Evaluation:
python -m evidencedesk.retrieval.benchmark --gate | tail -1GATE: GOWrite the operations runbook
| Incident | First check | Fix | Verify, roll back |
|---|---|---|---|
| Stale answer | stale_report, conflicts, cached entry age | fix or re-ingest the sibling; clear the cache | regression suite; rollback to the previous index |
| Missing document | explain (operator only) | fix the quarantine reason, ACL or listing | search as the user; verify |
| ACL drift or leak | verify (acl_drift), late denials | re-run the listing; narrow first, investigate second | leak tests of the previous page |
| Failed or partial run | verify, full-rebuild audit | re-run the same listing | staged copy, then cutover |
| Bad cutover | gate result, shadow disagreements | rollback while the old index is kept | citation resolve on stored answers |
explain names documents a user may not know exist, so it is for operators only; users always get "No matching documents." Five incidents:
for who, sid, q in ((CUSTOMER, "kc-stores", ""), (STAFF, "kc-goodwill", ""), (CUSTOMER, "kc-promo", ""), (HARBOR, "kc-returns", "return window"), (CUSTOMER, "kc-warranty", "return a bike")):
print(f"{sid:<12}for {who.user_id:<7}", ops.explain(kb, who, sid, q))kc-stores for cust-7 ['deleted: tombstone (removed_from_source)']
kc-goodwill for aiko ["acl denies: needs one of ['group:team-leads']"]
kc-promo for cust-7 ['quarantined at clean: hidden_text']
kc-returns for omar ['other tenant']
kc-warranty for cust-7 ['readable and indexed', "ranked below 3 for the query; top: ['kc-returns', 'kc-returns-faq', 'kc-shipping']"]Mini-lab: replay a content change
One command replays the Monday change: reconcile, staged run, gate, shadow comparison, cutover, citation redirects, audit and rollback. The wall-clock seconds of the tests are stripped with sed.
python -m evidencedesk.ingest.opslabday 0 live v1: 19 chunks, baseline evidence_at_3 8/16
day 6 reconcile finds: {'kc-recall-notice': 'new', 'kc-returns': 'changed', 'kc-stores': 'vanished'}
stage v2: verify 0 findings, gate BLOCK (critical case r16 failed; floor evidence_at_3 0.812 (13/16); floor stale_free 0.667 (2/3); flips q07)
why: kestrel/return_window_days: kc-returns=21, kc-returns-bn=14 -> prefer kc-returns (by recency), flag kc-returns-bn
stage v3: verify 0 findings, open conflicts 0, gate REVIEW (flips q07)
a person reviews the flip q07 (a paraphrase miss) and approves; it goes to the backlog
shadow: hit 0.6 -> 0.933, better 6 worse 1, same first source 0.8, embedded 6
cutover: citations the old answers stored now resolve to
kc-returns:c0 revised kc-returns v2, Return window, lines 2-3
kc-returns:c1 current kc-returns v2, Exceptions, line 6
kc-stores:c0 gone This passage no longer exists.
kc-goodwill:c0 gone This passage no longer exists.
audit: 8 of 8 checks ok
rollback: live serves 19 chunks again (v1)python -m pytest tests/test_retops_fresh.py tests/test_retops_ids.py tests/test_retops_shadow.py \
tests/test_retops_cache.py tests/test_retops_gate.py tests/test_retops_mining.py tests/test_retops_text.py \
| sed 's/ in [0-9.]*s//' .......................................... [100%]
42 passedWhere this shows up at work
Within a quarter the questions change from "why is this ranked second" to "why did the assistant quote last year's policy", "did the new embedding model make things worse" and "can we roll back before the weekend". The gate result, the shadow comparison and the runbook are what you show the page owner and the person on call.
Common mistakes
- Treating age as staleness. Old pages can be right. Use age for a review queue and conflicts to find the wrong ones.
- Merging contradictory evidence. No policy says 17. Resolve by rule or show both with provenance.
- Using the fused score as a confidence. It is rank-based. Calibrate another signal on labelled cases and report the sample size.
- Cutting over without a shadow read or a rollback. Compare on a fixed set first and keep the old index until the window closes.
- Citing
source:cN. Store the section anchor and the text hash, and resolve through permissions.
Try it yourself
- Easy (revisits Prompt Injection and Access Boundaries): after snapshot v2 narrows the goodwill note to team leads, Aiko (staff) opens her stored citation of it, and Omar (Harbor) opens a citation that never existed. Predict both answers.
- Medium (revisits Budgets, Caching, and Usage Limits): in the v2 state, Aiko asks about the goodwill limit through a
TagCache; then the note is opened to staff again. Predict whethersynckeeps the cache correct, and fix it in one line. - Hard (the lab, applied): in a temporary copy of the files, change the refund time in the English returns page from 5 to 3 business days. Predict which conflict appears and whether it can be resolved, then fix the Bangla page and show it closes.
Answers
Easy. Both are gone with the same message, so the citation is not an oracle for "this document exists".
kb_e, _ = build("v1")
goodwill = stableid.cite(kb_e, "kc-goodwill:c0")
kb_e.ingest(snapshot("v2"), prune=True) # the note is now readable by team leads only
ghost = stableid.Cite("kc-ghost", "kc-ghost:c0", "nothing", 1, "00000000")
a, b = stableid.resolve(kb_e, STAFF, goodwill), stableid.resolve(kb_e, HARBOR, ghost)
print(a.status, b.status, a == b)gone gone TruePass checklist: identical results; no title, source or version in the message.
Medium. No: sync only drops entries whose tags are touched, and widening access adds a result no tag mentions. Clear the cache (or put a permission version in the key) when any outcome is acl_updated.
from evidencedesk.ingest.access import search
kb_m, _ = build("v2")
cache = TagCache(kb_m)
Q = "goodwill credit limit"
cache.ask(STAFF, Q) # staff cannot read the note yet, so the cached answer holds other chunks
wide = [dict(e, allow=["group:staff"]) if e["source_id"] == "kc-goodwill" else e for e in snapshot("v2")]
report = kb_m.ingest(wide, prune=True)
print("dropped", cache.sync(report), "| stale / served from cache:", stale_hits(cache, kb_m, [(STAFF, Q)]))
cache.cache.clear() # the fix: a permission change can ADD results, so clear (or put a permission version in the key)
print("after the fix:", stale_hits(cache, kb_m, [(STAFF, Q)]), [h.source_id for h in search(kb_m, STAFF, Q).hits][:1])dropped 0 | stale / served from cache: (1, 1)
after the fix: (0, 0) ['kc-goodwill']Pass checklist: you saw one stale hit before the fix and none after; you measured stale hits, not the hit rate.
Hard. The English page now says 3 and the Bangla page 5, both changed on the same fake day, so the conflict is open; editing the Bangla digits closes it.
import shutil
import tempfile
from pathlib import Path
from evidencedesk.ingest.contract import DATA
with tempfile.TemporaryDirectory() as folder:
work = Path(folder)
shutil.copytree(DATA / "files", work / "files")
kb_h = KnowledgeBase(root=work)
kb_h.ingest(snapshot("v3"))
page = work / "files/kc-returns.v2.md"
page.write_text(page.read_text(encoding="utf-8").replace("within 5 business days", "within 3 business days"), encoding="utf-8")
kb_h.ingest(snapshot("v3"))
for c in conflict.conflicts(kb_h):
print(conflict.resolve(kb_h, c).text())
bn = work / "files/kc-returns-bn.v2.md"
bn.write_text(bn.read_text(encoding="utf-8").replace("৫ কার্যদিবসের", "৩ কার্যদিবসের"), encoding="utf-8")
kb_h.ingest(snapshot("v3"))
print(conflict.conflicts(kb_h))kestrel/refund_business_days: kc-returns=3, kc-returns-bn=5 -> UNRESOLVED, show all with provenance
[]Pass checklist: the conflict names both sources and values; it is unresolved, not guessed; the second check is empty.
Summary
- Detect staleness by age per origin, a cheap reconcile, held-back files and stale siblings; resolve contradictions by authority, then recency, else show both with provenance.
- Test long documents (cut in two or lost in top-k) and multilingual retrieval by slice and k; design fallbacks with a threshold calibrated on labelled cases.
- Count a run's cost before pricing it, debounce events and reconcile nightly, and change the model or chunker behind a shadow index with a rollback.
- Citations carry anchor and text hash; caches are judged by stale hits; audits compare the index with a full rebuild; recovery is a repeatable job on a staged copy.
- Mine failures into a labelled backlog, gate every re-index with a regression suite, and keep a runbook for operators.
Next: Capstone Architecture and Integration. The next page chooses a narrow production use case and connects retrieval, tools, evaluation, security and operations into one system.