Ingestion and Permission-Aware Retrieval

Mateo in Bogotá re-imports the help-centre export on Monday. By Tuesday a customer chat quotes a sentence from an internal pay note, with a citation. Nothing crashed and the answer was well formed. The mistake happened before retrieval: the note entered the index with no owner and no access rule, and search did what it was told, which was to search everything. (An illustrative case.)

The earlier pages improved ranking: Hybrid Retrieval and Error Analysis and Reranking and Retrieval Evaluation. This page is about what gets ranked, and for whom. It builds a governed ingestion pipeline and a permission-aware search in a new subpackage, evidencedesk/ingest/, on the Beginner chunker, index and BM25 and the Advanced retrieval package. Honest limits: the embedder is still the toy of Embeddings and Similarity, every document is synthetic, and the access model is simplified; a later section says what real systems add.

What you will learn

• How to write an ingestion contract and run it as stages that quarantine bad or hostile files with a reason instead of half-indexing them.
• How to make a re-run idempotent, keep versions and provenance, and handle corrections, deletions and erasure.
• How to re-index only what changed, and what "incremental" cannot save.
• How to filter by permission before ranking, keep user filters apart from security filters, and test for leaks through results, snippets, scores, errors and caches.
• How to evaluate query rewriting and context compression on a fixed set before adopting them.

Define an ingestion contract

A contract says what may enter, what must travel with it, and what happens to a file that breaks the rules. Formats: markdown, HTML, CSV, JSON and text (PDF text someone already extracted). Metadata, with no defaults because a default is a decision nobody made: source id, tenant, owner, ACL (who may read), sensitivity, licence, origin. Validation: size, encoding, structure, tenant and ACL rules, and screens for hidden text, instructions aimed at the assistant, and secret- or personal-data-shaped strings. Failure handling: a failing file goes to quarantine with a reason code and the stage that refused it, and the last good version, if any, keeps serving.

# excerpt of kit/evidencedesk/ingest/contract.py
class Source(BaseModel):
    """One entry of a snapshot: the file and everything the file itself cannot say. No default for tenant, owner or licence."""

    model_config = ConfigDict(extra="forbid", frozen=True)

    source_id: str = Field(pattern=r"^[a-z0-9][a-z0-9-]{2,39}$")
    path: str = Field(min_length=1)
    tenant: str = Field(min_length=1)
    owner: str = Field(min_length=1)
    allow: tuple[str, ...]
    sensitivity: Literal["public", "internal", "restricted"]
    licence: str = Field(min_length=1)
    origin: str = Field(min_length=1)
    lang: Literal["en", "bn"] = "en"

Source files sit in data/ingest/files/; a snapshot lists them, one JSON entry per file. The first has twelve entries in five formats and two companies: Kestrel Cycles, and Harbor Bikes, a second tenant that shares the knowledge base.

from evidencedesk.ingest.lab import snapshot

for e in snapshot("v1"):
    print(f"{e['source_id']:<16}{e['path'].rsplit('.', 1)[1]:<5}{e['tenant']:<8}{e['sensitivity']:<11}{e['allow'][0]}")

kc-returns      md   kestrel public     tenant:kestrel
kc-returns-faq  md   kestrel public     tenant:kestrel
kc-returns-bn   md   kestrel public     tenant:kestrel
kc-shipping     html kestrel public     tenant:kestrel
kc-warranty     csv  kestrel public     tenant:kestrel
kc-stores       json kestrel public     tenant:kestrel
kc-fleet-policy txt  kestrel internal   group:staff
kc-goodwill     md   kestrel internal   group:staff
kc-payroll-note md   kestrel restricted group:hr
hb-returns      md   harbor  public     tenant:harbor
hb-recall       md   harbor  internal   group:staff
kc-promo        html kestrel public     tenant:kestrel

A document leaves the eight stages at the first one that refuses it:

discover   entry valid? format accepted? file present and small enough?
parse      strict UTF-8, no NUL bytes; every format -> title + '## ' sections + '|' tables
clean      NFC, no invisible characters; SCREEN hidden text, injection words, secret and personal-data shapes
classify   tenant known? ACL well formed? label not looser than the content suggests?
chunk      structure-aware chunks (tables keep their header, code is never cut)
dedupe     identical text is indexed once inside one tenant AND one ACL
index      one partition per tenant: BM25 + vectors; each chunk carries its ACL
verify     do manifests, chunk records and index copies still agree?

Parse, normalise and clean

Every format becomes one shape, so one chunker serves all. HTML loses navigation, scripts and footers and keeps headings and tables; CSV and JSON become pipe tables. The text file shows the usual PDF damage: hard-wrapped lines, a word split at a line end, page furniture. fix_pdf_breaks drops lines that repeat three times or look like page numbers, joins wrapped lines, joins a hyphenated word only before a lower-case letter, and turns numbered lines into headings:

from evidencedesk.ingest.contract import DATA
from evidencedesk.ingest.parse import fix_pdf_breaks, parse

raw = (DATA / "files/kc-fleet-policy.txt").read_text(encoding="utf-8")
print("\n".join(raw.splitlines()[:5]))
print("-- rebuilt --")
print(fix_pdf_breaks(raw).split("\n\n")[:2])
page = parse(".html", (DATA / "files/kc-shipping.html").read_bytes(), "x")
print("-- html --", page.title, page.findings, "|", page.text.splitlines()[0], "|", page.text.splitlines()[4])
print("-- hostile html --", parse(".html", (DATA / "files/kc-promo.html").read_bytes(), "x").findings)

KESTREL CYCLES - INTERNAL
1. Demo bike lending
Staff may borrow a demo bike for up to three days. The borrower signs the bike out and
returns it with a full char-
ge, clean, with all accessories.
-- rebuilt --
['## Demo bike lending', 'Staff may borrow a demo bike for up to three days. The borrower signs the bike out and returns it with a full charge, clean, with all accessories.']
-- html -- Shipping and delivery () | ## Delivery times | | Speed | Days | Cost |
-- hostile html -- ('hidden_text',)

The hostile page's visible text is harmless; the danger is a display:none element and a comment, which readers never see and extractors read happily. The parser does not put hidden text into the document; it reports it and the pipeline quarantines the file. Joining at a hyphen is a heuristic: e- then bike becomes ebike (a test pins that failure). clean applies NFC, removes zero-width, direction-override and control characters and boilerplate lines, and keeps the joiners inside Bangla conjuncts, because removing them changes spelling.

Classify: sensitivity, tenant and audience

The ACL is a set of tokens: tenant:kestrel (everyone in that company, customers included), group:staff, user:nadia. The caller is a Principal supplied by the login system; its tokens decide, and nothing in the query text can add one. An empty ACL means nobody, so a forgotten field fails closed.

# excerpt of kit/evidencedesk/ingest/contract.py
    def permits(self, who: Principal) -> bool:
        return self.tenant == who.tenant and bool(self.allow & who.tokens)

    def tokens(self) -> frozenset[str]:
        return frozenset({f"user:{self.user_id}", f"tenant:{self.tenant}", *(f"group:{g}" for g in self.groups)})

classify refuses guesses. An unknown tenant, an unparsable token, or tenant: on internal or restricted content is quarantined, and so is content that looks more sensitive than its label (the toy cue list finds "payroll" in a file declared public). It never loosens a label and never invents a tenant; a person decides.

Build an idempotent ingestion job

Idempotent: run the same input twice and the second run changes nothing. The key is a hash of the cleaned text (a re-export with new whitespace is not a new version) plus ACL and metadata, compared with the active manifest:

# excerpt of kit/evidencedesk/ingest/kb.py
        cur = self.manifests.get(sid)
        cur = cur if cur and cur.status == "active" else None
        meta = dict(owner=src.owner, licence=src.licence, origin=src.origin, lang=src.lang, title=title, fmt=fmt)
        if cur and cur.content_hash == digest:  # same content: at most the permissions or the metadata changed
            same_meta = all(getattr(cur, k) == v for k, v in meta.items())
            if cur.acl == acl and same_meta:
                return Outcome(sid, "unchanged", cur.version, len(cur.chunk_ids))
            return self._swap(cur, replace(cur, acl=acl, ingested_at=now, run_id=run_id, **meta), "acl_updated" if cur.acl != acl else "meta_updated")

Outcomes: unchanged (nothing written), acl_updated or meta_updated (same text: no chunking, no embedding), updated (new version) and added. A quarantined file is recorded once per distinct refusal, so a re-run does not grow the history. Ingest the first snapshot twice:

from evidencedesk.ingest.lab import fresh, snapshot

kb, report = fresh("v1")
print(report.text())
again = kb.ingest(snapshot("v1"))
print("second run:", again.counts(), "| embedded", again.embedded, "| epoch", again.epoch, "| history rows", len(kb.history))

run-001  epoch 1  embedded 18  verify findings 0
  hb-recall        added        v1 chunks 1
  hb-returns       added        v1 chunks 1
  kc-fleet-policy  added        v1 chunks 3
  kc-goodwill      added        v1 chunks 2
  kc-payroll-note  added        v1 chunks 1
  kc-promo         quarantined  v0 chunks 0  [clean] hidden_text
  kc-returns       added        v1 chunks 2
  kc-returns-bn    added        v1 chunks 1
  kc-returns-faq   added        v1 chunks 2
  kc-shipping      added        v1 chunks 3
  kc-stores        added        v1 chunks 1
  kc-warranty      added        v1 chunks 2
second run: {'quarantined': 1, 'unchanged': 11} | embedded 0 | epoch 1 | history rows 12

The second run embeds nothing, keeps the epoch (the change counter the cache keys use) and adds no history rows. The hostile page is refused again; the run number advances because a run happened.

Track document versions and provenance

Each revision gets a manifest: metadata only, no text. content_hash, version and run_id say which bytes, which revision, which run; licence and origin say whether we may use it and where it came from; stages lists what completed. Ingest time comes from an injected fake clock.

manifest = kb.manifests["kc-returns"].as_dict()
for key in ("source_id", "version", "status", "content_hash", "tenant", "owner", "acl", "licence", "origin", "ingested_at", "run_id", "chunk_ids", "stages"):
    print(f"{key:<13}{manifest[key]}")

source_id    kc-returns
version      1
status       active
content_hash 7f871158317e
tenant       kestrel
owner        returns-team
acl          {'tenant': 'kestrel', 'allow': ['tenant:kestrel'], 'sensitivity': 'public'}
licence      synthetic-course-data
origin       help-centre-export
ingested_at  1000.0
run_id       run-001
chunk_ids    ['kc-returns:c0', 'kc-returns:c1']
stages       ['discover', 'parse', 'clean', 'classify', 'chunk', 'dedupe', 'index']

Now the second snapshot: one policy changed, one note's permissions changed, one file vanished, one is new.

v2 = kb.ingest(snapshot("v2"), prune=True)
print(v2.text().splitlines()[0])
for o in v2.outcomes:
    if o.outcome != "unchanged":
        print(f"  {o.source_id:<17}{o.outcome:<12}v{o.version} {'; '.join(o.reasons)}")
print([(m.version, m.status, m.run_id) for m in kb.history if m.source_id == "kc-returns"])

run-003  epoch 3  embedded 2  verify findings 0
  kc-returns       updated     v2 
  kc-goodwill      acl_updated v1 
  kc-promo         quarantined v0 hidden_text
  kc-recall-notice added       v1 
  kc-stores        deleted     v1 removed_from_source
[(1, 'active', 'run-001'), (1, 'superseded', 'run-001'), (2, 'active', 'run-003')]

That is handling a correction: kc-returns became version 2, its old chunks left the index, and the history keeps a superseded record. Each chunk record links to its revision and run:

from evidencedesk.ingest.access import search
from evidencedesk.ingest.lab import CUSTOMER

rec = kb.records["kc-returns:c0"]
print(rec.chunk.id, "| source", rec.source_id, "v%d" % rec.version, "| ingested in", rec.run_id, "| chunk hash", rec.chunk_hash, "| acl", sorted(rec.acl.allow))
print("citation:", search(kb, CUSTOMER, "How long do I have to return a bike?", 1).hits[0].citation)

kc-returns:c0 | source kc-returns v2 | ingested in run-003 | chunk hash 7235da72 | acl ['tenant:kestrel']
citation: kc-returns v2, Return window, lines 2-3

Two honest notes. Citation line numbers refer to the cleaned text, not the original HTML or PDF, so the manifest hash is what ties a citation to a source revision. And chunk ids are source:cN, so kc-returns:c0 now names different text than last week; whether an old citation still resolves is the subject of Retrieval Operations and Data Freshness.

Quarantine hostile and messy sources

The repository holds one hostile file. The other bait is generated on demand (bait.write_messy), so no scanner trips on it. Sixteen bad sources and one untidy one, printed by the stage that handled them:

import tempfile
from pathlib import Path

from evidencedesk.ingest import bait
from evidencedesk.ingest.kb import KnowledgeBase

with tempfile.TemporaryDirectory() as folder:
    messy = KnowledgeBase(root=Path(folder))
    result = messy.ingest(bait.write_messy(Path(folder)))
for o in sorted(result.outcomes, key=lambda o: (o.stage, o.source_id)):
    print(f"{o.stage or 'ok':<9}{o.source_id:<13}{'; '.join(o.reasons)}")

ok       tidy-me      
classify mislabelled  sensitivity_mismatch:restricted
classify no-acl       acl_empty
classify other-tenant unknown_tenant; acl_too_broad
classify wide-open    acl_too_broad
clean    blank        empty
clean    has-contact  personal_data:email
clean    has-key      secret_shaped
clean    shouting     injection_signals:override,secret_request
discover gone         missing_file
discover huge         too_large
discover no-tenant    bad_metadata:tenant
discover tool-exe     unsupported_format
parse    bad-json     bad_json
parse    latin1       bad_encoding
parse    nul-bytes    binary
parse    ragged       ragged_table

Each reason is checkable and none contains the text. The screens are blunt: the Beginner injection signals, key shape and personal-data shapes catch the careless case, and Safe Failure and Security Regression Tests says why an empty screen is not proof of safety. A legitimate document that quotes a phishing example would be held too; quarantine is a review queue, not a verdict.

Handle deletions and erasure

A superseded version is one thing, a deletion another. delete removes the chunks, rebuilds the tenant's index, drops cached vectors nobody uses, invalidates cached answers that cite the source, and rewrites the source's history rows so they keep no title, hash or owner. What stays is a tombstone: source id, tenant, the fact and the reason. A prune run turns a vanished source into a deletion, but refuses when that would remove more than half of the active sources or the listing is empty, because a failed export looks exactly like "delete everything". An erasure counts only when something checks every copy, so the kit has that check; the example proves it can see the copies before the deletion:

from collections import Counter

from evidencedesk.ingest.access import CachedSearch
from evidencedesk.ingest.lab import LEAD
from evidencedesk.ingest.verify import erasure_residue, remember

cache = CachedSearch(kb)
cache.ask(LEAD, "goodwill credit limit")  # an answer that quotes the source is now in the response cache
saved = remember(kb, "kc-goodwill")
print("before:", dict(sorted(Counter(r.split(":")[0] for r in erasure_residue(kb, saved)).items())))
kb.delete("kc-goodwill", "erasure request 12")
print("after: ", erasure_residue(kb, saved), "| tombstone:", kb.manifests["kc-goodwill"].status, kb.manifests["kc-goodwill"].reasons)

before: {'cache0': 1, 'history': 3, 'index': 2, 'manifest': 1, 'record': 2, 'vector': 2}
after:  [] | tombstone: deleted ('erasure request 12',)

The check covers records, history, manifests, index, embedding cache and response cache. It cannot see backups, logs shipped elsewhere, replicas or a fine-tuned model; a real procedure names those and gives each an owner and a date.

Design incremental re-indexing

The run for the second snapshot embedded two chunks, although the first run embedded eighteen. Embeddings are cached by the hash of the embedded text, so unchanged text is never sent to the model again, and a permission-only change embeds nothing and does not rebuild the search structures. What is not incremental: after a content change the tenant's BM25 statistics and vector matrix are rebuilt, because IDF depends on the whole partition. That is cheap here and wrong at a million chunks, where you use an engine that upserts and deletes points. The claim is "reprocess changed documents, re-embed changed text".

Duplicates need one rule: identical chunks are indexed once only when tenant and ACL are equal. A public line and an identical staff-only line must both exist, or a public caller loses a result, or a citation reveals a staff document. When the canonical copy's source is deleted, the next copy is promoted, so deleting one document never deletes text another still holds.

Filter before you rank

This is the central rule, and the Beginner OwnedStore of Prompt Injection and Access Boundaries already follows it. The security filter is a set of chunk ids computed from the caller's identity before any scoring, inside the caller's tenant partition, so another tenant's vectors and BM25 statistics are never touched:

# excerpt of kit/evidencedesk/ingest/access.py
def rank(kb: KnowledgeBase, who: Principal, query: str, k: int = 3, filters: Filters | None = None) -> tuple[RankedList, int]:
    """(candidates, late denials). The one place where retrieval and permissions meet."""
    allowed = narrow(kb, security_allowed(kb, who), filters)  # 1. filter first
    part = kb.parts.get(who.tenant)
    found = part.hybrid.search(query, k, allowed) if allowed and part and part.hybrid else []  # 2. rank only what is allowed
    kept = [c for c in found if (m := kb.manifests.get(c.chunk.doc_id)) and m.status == "active" and m.acl and m.acl.permits(who)]
    # 3. defence in depth: re-check the live manifest, because the index copy of an ACL can be stale
    return kept, len(found) - len(kept)

Step 3 is defence in depth: the index keeps its own copy of each ACL (what a vector database calls a payload), and a copy can go stale. Now the anti-pattern, post-filtering: rank everything, cut to k, then remove what the caller may not see. Omar, a Harbor customer, asks one question:

from evidencedesk.ingest import naive
from evidencedesk.ingest.lab import HARBOR

shared = naive.shared_index(kb)  # the anti-pattern: one index for every tenant
q = "Who pays for repairs after damage?"
post, total = naive.post_filter_search(kb, shared, HARBOR, q, 3)
print("post-filter:", post, "| 'matches' shown to Omar:", total)
print("pre-filter: ", [h.chunk_id for h in search(kb, HARBOR, q, 3).hits])
print("user dict wins:", naive.merged_filter_search(kb, shared, HARBOR, "return a bike", {"tenant": "kestrel"}))

post-filter: [] | 'matches' shown to Omar: 16
pre-filter:  ['hb-returns:c0']
user dict wins: ['kc-returns:c0', 'kc-returns-faq:c1', 'kc-fleet-policy:c0']

The post-filter returns nothing because the top three chunks all belong to Kestrel; the pre-filter returns Harbor's own answer. Worse, the "matches" figure counts chunks Omar must not know exist. The last line is the second anti-pattern, and the reason separate user filters from security filters is its own rule: when both are one dict merged with the user's keys last, a request that says tenant: kestrel wins. The typed filter has no field for tenant, ACL or sensitivity, forbids unknown fields and can only narrow, by intersection:

from pydantic import ValidationError

from evidencedesk.ingest.access import Filters

for bad in ({"tenant": "kestrel"}, {"allow": ["tenant:kestrel"]}):
    try:
        Filters(**bad)
    except ValidationError as error:
        print("refused:", error.errors()[0]["type"], error.errors()[0]["loc"])
asked = search(kb, HARBOR, "return a bike", 5, Filters(sources=("kc-returns",)))
print("Omar names a Kestrel source:", asked.hits, repr(asked.message))

refused: extra_forbidden ('tenant',)
refused: extra_forbidden ('allow',)
Omar names a Kestrel source: () 'No matching documents.'

Naming a source you cannot read gives the same empty answer as naming one that does not exist, so the filter is not an oracle for "this document exists".

Provider note (as of 2026-10-12): the qdrant-client README shows query_points(..., query_filter=models.Filter(must=[models.FieldCondition(...)])): the filter is part of the search call, the same idea as above. Local mode runs exact brute-force search and warns above 20,000 points. A self-deployed Qdrant server accepts requests from anyone who can reach it unless an API key is enabled. Check the current documentation, and confirm when your own store applies a filter (before, during or after the nearest-neighbour step).

Security note (as of 2026-10-12): in the 2025 edition of the OWASP Top 10 for LLM Applications the entry "Vector and Embedding Weaknesses" lists unauthorized access and data leakage, and cross-context information leaks, among its risks. A 2026 edition renumbers entries, so cite risks by name and edition. Check the current documentation.

Test retrieval with permission changes

Permissions change after the index is built: a note is narrowed, a person changes team. Change the permission in the source and ask the same question again, through the cache too:

from evidencedesk.ingest.lab import STAFF

Q = "goodwill credit limit"


def goodwill_for(person):
    return sorted({h.source_id for h in search(kb1, person, Q, 3).hits if h.source_id == "kc-goodwill"})


kb1, _ = fresh("v1")
cached = CachedSearch(kb1)
print("v1  staff:", goodwill_for(STAFF), "| lead:", goodwill_for(LEAD), "| cache:", cached.ask(STAFF, Q)[1], cached.ask(STAFF, Q)[1])
change = kb1.ingest(snapshot("v2"), prune=True)  # the goodwill note now allows group:team-leads only
print("v2  staff:", goodwill_for(STAFF), "| lead:", goodwill_for(LEAD), "| cache:", cached.ask(STAFF, Q)[1])
print("goodwill:", {o.source_id: o.outcome for o in change.outcomes}["kc-goodwill"], "| no re-chunking for it: version", kb1.manifests["kc-goodwill"].version)

v1  staff: ['kc-goodwill'] | lead: ['kc-goodwill'] | cache: miss hit
v2  staff: [] | lead: ['kc-goodwill'] | cache: miss
goodwill: acl_updated | no re-chunking for it: version 1

After the second snapshot staff no longer see the goodwill note, the lead still does, and the cache did not replay the old answer, because its key carries the caller's scope and the knowledge base's epoch (scope keys come from Cost and Latency Fundamentals). And if an index copy of an ACL is wrong? Make one too loose on purpose:

from evidencedesk.ingest.contract import Acl
from evidencedesk.ingest.verify import verify

before = kb1.late_denied
kb1.parts["kestrel"].acl["kc-payroll-note:c0"] = Acl("kestrel", frozenset({"tenant:kestrel"}), "public")  # a stale, too-loose index copy
hits = search(kb1, CUSTOMER, "payroll cut-off salary", 5).hits
print("payroll chunk returned:", any(h.source_id == "kc-payroll-note" for h in hits), "| late denials:", kb1.late_denied - before)
print("verify:", verify(kb1))
kb1.ingest(snapshot("v2"), prune=True)  # the sources did not change, yet index copies are derived again
print("after a re-run:", verify(kb1))

payroll chunk returned: False | late denials: 1
verify: [Finding(code='acl_drift', subject='kc-payroll-note:c0')]
after a re-run: []

The payroll chunk was a candidate (the index copy allowed it) and the manifest re-check removed it. verify reports the drift and a re-run heals it, since index copies are derived from manifests on every run. Late denials should be near zero in production; alert when they are not.

Prove there is no leak

"The filter works" is not a test. A leak is any place where one person's answer carries information about data they may not see. The tests ask five people the same nine questions and check every channel:

only_kestrel = [e for e in snapshot("v1") if e["tenant"] == "kestrel"]
a, b = KnowledgeBase(), KnowledgeBase()
a.ingest(only_kestrel)
b.ingest(snapshot("v1"))  # the same plus Harbor's documents
q = "How long do I have to return a bike?"
print("scores identical with and without the other tenant:", search(a, CUSTOMER, q, 5) == search(b, CUSTOMER, q, 5))
c = KnowledgeBase()
c.ingest([e for e in only_kestrel if e["source_id"] != "kc-payroll-note"])
raw = lambda base: round(base.parts["kestrel"].hybrid.sparse.bm25.score(q, "kc-returns:c0"), 4)
print("raw BM25 of one chunk with / without a hidden HR note:", raw(a), raw(c))

scores identical with and without the other tenant: True
raw BM25 of one chunk with / without a hidden HR note: 1.8298 1.765

The second line is a residual leak, stated rather than hidden: inside a tenant, a hidden HR note moves the raw BM25 score of a public chunk. Rank-based fusion shields the fused score, but only mostly; give very sensitive content its own partition. The simplified model also lacks what real systems add: deny rules and inheritance, nested groups, permission sync from the source system (with a lag you must measure), per-request token exchange and an audit trail of reads. The four test files (wall-clock seconds stripped with sed):

python -m pytest tests/test_ingest_pipeline.py tests/test_ingest_access.py tests/test_ingest_erasure.py tests/test_ingest_rewrite.py | tail -n 1 | sed 's/ in [0-9.]*s//'

44 passed

Observe the pipeline

The knowledge base writes events through the logger of Structured Logging and Tracing: counts, outcome codes, a pseudonymous user, the query's length instead of the query, and no chunk text.

from evidencedesk.observe.events import EventLog
from evidencedesk.usage import ManualClock

log = EventLog(ManualClock(1000.0))
kb3, _ = fresh("v1", log=log)
search(kb3, CUSTOMER, "salary review and payroll", 3, request_id="req-1")
for e in log.records():
    if e["event"] in ("ingest.run_end", "search.query"):
        print(e["event"], e["user"] or "-", e["fields"])

ingest.run_end - {'embedded': 18, 'epoch': 1, 'findings': 0, 'n_added': 11, 'n_quarantined': 1, 'run_id': 'run-001'}
search.query u-d14b511a {'hits': 3, 'k': 3, 'late_denied': 0, 'query_chars': 25}

Watch outcome counts, quarantine reasons, embedded (the cost), verify findings and late denials (both should be 0).

Evaluate query rewriting carefully

A rewriter turns the user's question into a better search query, often with a model. It can help paraphrases and hurt identifiers. The method: fix the question set (data/ingest/questions.jsonl: who asks, the evidence phrase a returned chunk must hold, or a forbidden phrase it must not), score original and rewritten query on the same set, and compare question by question with the exact sign test of Reranking and Retrieval Evaluation. Our rewriters are deterministic stand-ins; a model call is not run here. The third tries to smuggle restricted words into the query:

from evidencedesk.ingest.rewrite import DOMAIN, StripRewriter, TableRewriter, compare, compress_report, load_cases, outcome

cases = load_cases()
print(len(cases), "fixed questions | hit@1 of the original questions:", round(sum(outcome(kb1, c, c.question, 1) for c in cases) / len(cases), 3))
for r in (DOMAIN, StripRewriter(), TableRewriter("greedy", (("return", "return salary reviews payroll goodwill"),))):
    got = compare(kb1, cases, r, 1)
    print(f"{r.name:<7} {got.original} -> {got.rewritten} | better {list(got.delta.better)} worse {list(got.delta.worse)} p={got.delta.p} | leaks {got.leaks}")

15 fixed questions | hit@1 of the original questions: 0.733
domain  0.733 -> 0.867 | better ['q05', 'q07'] worse [] p=0.5 | leaks 0
strip   0.733 -> 0.667 | better [] worse ['q15'] p=1.0 | leaks 0
greedy  0.733 -> 0.733 | better [] worse [] p=1.0 | leaks 0

The domain table fixes two paraphrases, 2 to 0, p = 0.5: encouraging, not proven, and its rules were written while reading these questions, so a fresh set would likely show less. The "cleaning" rewriter drops tokens with a digit or hyphen and loses RD2-24. The greedy rewriter changes the query and leaks nothing, because the security filter runs after the rewrite on the caller's identity. Never let a rewriter see or choose the filters.

Add context compression as an optional stage

Compression shrinks retrieved text before the prompt. The simplest kind is selection: keep the sentence of each chunk that shares most words with the question (pick_spans). It runs after retrieval on allowed text only, and must be measured for what it loses:

for per_chunk in (1, 2):
    got = compress_report(kb1, cases, per_chunk, 3)
    print(f"{per_chunk} sentence(s) per chunk: {got.tokens_before} -> {got.tokens_after} tokens ({1 - got.tokens_after / got.tokens_before:.0%} saved), evidence lost in {list(got.lost)}")

1 sentence(s) per chunk: 1107 -> 609 tokens (45% saved), evidence lost in ['q03', 'q12']
2 sentence(s) per chunk: 1107 -> 985 tokens (11% saved), evidence lost in ['q03']

One sentence per chunk saves 45% of the tokens and loses the evidence in two of the 13 answerable questions (for the refund question the best-matching sentence is not the one with the number); two sentences save 11% and lose one. Adopt compression only if the lost-evidence count on your own set is acceptable at the saving you need, and re-measure when chunking changes.

Where this shows up at work

A support team connects a shared drive to its assistant. Within a month the questions are "who can see this", "why is last year's policy cited", "delete this person's file" and "what changed in the last import". The contract, manifests, quarantine list and leak tests are what you hand a security reviewer; the run report is what the on-call engineer reads when an import goes wrong.

Common mistakes

• Filtering after ranking. It returns fewer than k, or none, with counts from other tenants. Filter first and keep a control that proves the unfiltered search leaks.
• One dict for security and user filters. Whoever writes last wins. Derive the security set from identity; give users a typed, narrowing-only filter.
• Calling "removed from the index" a deletion. Vectors, caches, history, logs and backups keep the content. List the copies and check them.
• Deduplicating across ACLs. Merging a public and a restricted copy hides a result or reveals a document. Merge inside one tenant and one ACL.
• Calling a re-run "the same" without testing it. Second run: zero embeds, no history growth, same epoch.

Try it yourself

1. Easy (revisits RAG Ingestion and Citation Flow): copy the kc-warranty entry, mistype the tenant as kestrl, and predict the outcome, stage and reason before running it.
2. Medium (revisits Prompt Injection and Access Boundaries): make the fleet policy readable only by group:workshop. Predict how many chunks are embedded, the outcome, and who can read it afterwards.
3. Hard (the exercise, incremental knowledge-base update): in a temporary copy of the files, change the battery warranty from 24 to 36 months, remove kc-stores from the listing, re-ingest with prune and verify the final search results: the battery question returns the new row, no store text is left, verify reports nothing, and exactly one chunk is embedded.

Answers

Easy. Quarantined at classify: the tenant is unknown, and the token tenant:kestrel no longer matches the mistyped tenant, so the ACL is "too broad" as well. Nothing was chunked.

entry = dict(next(e for e in snapshot("v1") if e["source_id"] == "kc-warranty"), tenant="kestrl")
print(KnowledgeBase().ingest([entry]).outcomes[0])

Outcome(source_id='kc-warranty', outcome='quarantined', version=0, chunks=0, stage='classify', reasons=('unknown_tenant', 'acl_too_broad'))

Pass checklist: you named the stage; you saw two rules fire; nothing reached the index.

Medium. Nothing is embedded and the outcome is acl_updated; Aiko (staff) loses access and a group:workshop member gains it.

from evidencedesk.ingest.contract import Principal

entries = [dict(e) for e in snapshot("v2")]
next(e for e in entries if e["source_id"] == "kc-fleet-policy")["allow"] = ["group:workshop"]
step = kb1.ingest(entries, prune=True)
shop = Principal("priya", "kestrel", ("workshop",))
print({o.source_id: o.outcome for o in step.outcomes if o.outcome != "unchanged"}, "| embedded", step.embedded)
for person in (STAFF, shop):
    print(person.user_id, sorted({h.source_id for h in search(kb1, person, "borrow a demo bike", 3).hits if h.source_id == "kc-fleet-policy"}))

{'kc-fleet-policy': 'acl_updated', 'kc-promo': 'quarantined'} | embedded 0
aiko []
priya ['kc-fleet-policy']

Pass checklist: embedded is 0; the version did not change; the old group lost access on the next search without a restart.

Hard.

import shutil
import tempfile
from pathlib import Path

from evidencedesk.ingest.contract import DATA

with tempfile.TemporaryDirectory() as folder:
    work = Path(folder)
    shutil.copytree(DATA / "files", work / "files")
    kb4 = KnowledgeBase(root=work)
    kb4.ingest(snapshot("v1"))
    warranty = work / "files/kc-warranty.csv"
    warranty.write_text(warranty.read_text(encoding="utf-8").replace("battery,24", "battery,36"), encoding="utf-8")
    step = kb4.ingest([e for e in snapshot("v1") if e["source_id"] != "kc-stores"], prune=True)  # kc-stores vanished
    print({o.source_id: o.outcome for o in step.outcomes if o.outcome != "unchanged"}, "| embedded", step.embedded, "| findings", step.findings)
    hit = search(kb4, CUSTOMER, "How many months is the battery warranty?", 1).hits[0]
    print(hit.chunk_id, "| row says 36:", "| battery | 36 |" in hit.snippet)
    stores = search(kb4, CUSTOMER, "Which store has a workshop in Dhaka?", 3).hits
    print([h.source_id for h in stores], "| Dhaka in text:", any("Dhaka" in h.snippet for h in stores), "| tombstone:", kb4.manifests["kc-stores"].status)

{'kc-warranty': 'updated', 'kc-promo': 'quarantined', 'kc-stores': 'deleted'} | embedded 1 | findings []
kc-warranty:c0 | row says 36: True
['kc-warranty', 'kc-warranty', 'kc-returns'] | Dhaka in text: False | tombstone: deleted

One chunk is embedded (the table chunk with the battery row; the file's other chunk is unchanged and its vector is reused), and the deleted source leaves no chunk. The store question gets weak, unrelated hits from the toy dense search, with no store text in them. Pass checklist: you checked final results, not only counts; the deletion reached the index; no verify findings.

Summary

• An ingestion contract fixes formats, metadata, validation and failure handling; failures go to quarantine with a stage and a reason, never into the index.
• Idempotence comes from hashing cleaned content plus ACL and comparing with the active manifest; test it with a second run that embeds and writes nothing.
• Manifests and chunk records give provenance; corrections make a new version, deletions leave a tombstone, and an erasure is proven by a check over every copy.
• Re-embed only changed text, rebuild structures per tenant, say what is not incremental, and deduplicate only inside one tenant and one ACL.
• Filter by permission before ranking, from identity, with a typed narrowing-only user filter, a late re-check and one message for every empty answer. Test results, snippets, scores, errors, caches and logs.
• Query rewriting and context compression are optional: compare them with the original on a fixed set, behind the security filter.

Next: Retrieval Operations and Data Freshness. The next page builds on this pipeline: long documents, multilingual queries, fallback and contradiction policies, drift, citation stability across revisions, recovery from failed ingestion and the operations runbook.