Capstone, Cheat Sheet and What Next

Nadia in Dhaka has built a project over twenty pages. Her team lead asks three questions: does it run on a clean machine, does it do what the brief said, and what does it not do? A project is finished when someone else can answer them in ten minutes without asking her. This page is those ten minutes: the brief, a script that checks it, one page of commands, a map of the system, the failures you will meet, and the road ahead.

Everything runs offline. The model is a small stand-in, not a language model; real models are less predictable, which is exactly why you test. The page adds one small folder, capstone/, and one test file, and edits nothing that exists. The graded exercises are on Practice Lab; this page does not repeat them.

What you will learn

• The capstone brief, its five milestones and the exact earlier pages and kit modules behind each, with an acceptance check you can run for every milestone.
• How the grading rubric maps to evidence already in the kit, and how an acceptance checklist becomes a script.
• A one-page cheat sheet: commands, an architecture summary with trust boundaries, and a troubleshooting checklist built from failures reproduced for real.
• What Beginner left out on purpose, the honest limits of the kit, and the path into the Advanced tutorial.

The capstone brief

Goal. A compact assistant that answers questions about a small set of local documents, cites its evidence and abstains when it cannot find an answer. The user is a customer of the fictional Kestrel Cycles; the data is synthetic. Every required feature is already in the kit, so the capstone is assembling and proving, not writing new code. The pages that taught each one are named with the milestones below.

Provider note (as of 2026-10-12): the brief asks for two optional adapters chosen from OpenAI, Anthropic and Google Gemini. Google documents two API paths: the newer Interactions API is presented first, and generateContent is called legacy but still supported. Anthropic has no first-party embeddings, so its adapter covers generation and retrieval stays local. An adapter follows the four jobs in docs/INTERFACE.md and passes the adapter tests; none was run for this tutorial. Check the current documentation before you write one.

The five milestones, and how to prove each one

A milestone is done when a check says so, not when you feel finished. Each row names the pages that taught the skill, the modules involved and the check in capstone/accept.py (the checks M1 to M5 below).

Three required features cut across the milestones: configuration (Adapters, Configuration, and Error Handling), streaming (Streaming and Structured Results, Reliable Output Contracts) and cost control (Cost and Latency Fundamentals, Budgets, Caching, and Usage Limits). Their evidence is A1 (offline configuration) and the tests, for example tests/test_streaming.py and tests/test_limits.py.

The acceptance checklist as a script

The brief ends with six items: fresh-environment setup works; tests run without credentials; known questions retrieve the expected evidence; unsupported questions abstain; invalid tool calls are rejected; limitations are documented. capstone/accept.py turns each into a check (A1 to A6) using only existing modules: the release checks, the evaluation data, the in-process API and the abuse set. Two checks decide what "pass" means, so read them.

# excerpt of kit/capstone/accept.py
GAPS = {"rc15": "answered by the service: secure_answer applies only the min_score threshold, not the margin check that guarded_answer (and the eval) use"}


def a3_known(listed: str | None = None) -> Result:
    retriever = build_retriever(KIT / "corpus")
    listed = (KIT / "docs" / "EVALUATION.md").read_text(encoding="utf-8") if listed is None else listed
    cases = [c for c in load_rag_cases(KIT / "data" / "rag_cases.jsonl") if c.answerable]
    found, missed = [], []
    for c in cases:
        (found if set(c.expected_doc_ids) <= {r.chunk.doc_id for r in retriever.retrieve(c.question)} else missed).append(c.id)
    undocumented = [i for i in missed if f"| {i} |" not in listed]
    return _ok(not undocumented), f"{len(found)} of {len(cases)} retrieve the expected document; misses {', '.join(missed)} are listed in EVALUATION.md" \
        if not undocumented else f"undocumented misses: {undocumented}"


def a4_abstain() -> Result:
    cases = [c for c in load_rag_cases(KIT / "data" / "rag_cases.jsonl") if not c.answerable]
    answered = sorted(c.id for c in cases if _ask(c.question)["status"] != "refused")
    detail = f"{len(cases) - len(answered)} of {len(cases)} unanswerable questions refused"
    if answered != sorted(GAPS):
        return "FAIL", f"{detail}; answered {answered}, listed gaps {sorted(GAPS)}"
    return ("gap", f"{detail}; known gap {', '.join(GAPS)}") if GAPS else ("ok", detail)

a3_known allows a miss only if docs/EVALUATION.md already lists it, so a new, unreported failure fails the check. a4_abstain does the same for a gap that is known on purpose: GAPS must match reality in both directions, so a fixed gap makes the check fail until the entry is removed. Run the whole checklist.

python -m capstone.accept

CAPSTONE ACCEPTANCE evidencedesk
[ok  ] M1 document schema and acceptance criteria      13 documents, unique ids, status and lang valid
[ok  ] M2 fake client and local retrieval baseline     fake client answers; local search picks kc-returns-v3
[ok  ] M3 context, answer and citation validation      cited ['kc-returns-v3:c0'] resolve and support the claim; an invented id is rejected
[ok  ] M4 tool validation and failure handling         4 of 4 bad calls rejected, one good call runs
[ok  ] M5 evaluation, packaging and report             10 release checks pass (suite not run here)
[ok  ] A1 fresh-environment setup works                version, exact pins, lock file, data, offline default
[ok  ] A2 tests run without credentials                defaults need no key (pass --tests to run the suite without any key variable)
[ok  ] A3 known questions retrieve expected evidence   10 of 13 retrieve the expected document; misses rc03, rc06, rc08 are listed in EVALUATION.md
[gap ] A4 unsupported questions abstain                2 of 3 unanswerable questions refused; known gap rc15
[ok  ] A5 invalid tool calls are rejected              7 of 7 tool-misuse cases of the abuse set end as expected
[ok  ] A6 limitations are documented                   LIMITATIONS.md names 6 of 6 required topics
RESULT: ACCEPTED (0 failed, 1 known gap)

Read the gap line. It is a real finding from writing this page. The question "Do you sell electric scooters?" is answered with a sentence about battery charging: the service applies only the score threshold, while the evaluation uses a stricter pipeline that also checks the margin between the top two results. The numbers in docs/EVALUATION.md are true for the pipeline measured, not for the one users reach, and an all-green checklist would have hidden that. The exercises close the gap and show why fixing it from one case is risky. In A3, 10 of 13 answerable questions retrieve the expected document; rc13 is retrieved but refused for ambiguity, so the evaluation lists four English misses and this retrieval-only check three.

What the script cannot show: that answers are good. The model is a fake, thirteen questions are a small sample, and "limitations are documented" only checks that six topics are named. It guards against forgetting; judgement stays with a person.

The grading rubric and its evidence

The capstone is graded out of 100 by seven categories. A grade without evidence is an opinion, so each row names where the evidence is and what that evidence cannot show.

The weights add to 100. Tests, retrieval and correctness carry 60 of them, and the model itself is not a category: the course grades the engineering around a model, because that is what you control.

Cheat sheet: the commands

Run everything from the project folder (the one with pyproject.toml). Packages are pinned in requirements.txt; requirements.lock.txt lists the tested environment. Setup is the only step that needs the network, so it is not run here.

# not run here: creates an environment and downloads packages from the network
python3.12 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install -r requirements.txt

The same commands, run for real. The elapsed time is stripped from the test line with sed because it differs on every run.

python -m pytest | tail -1 | sed 's/ in [0-9.]*s//'
python -m evidencedesk.api
python -m capstone.accept | tail -1
python -m evidencedesk.release --tests | grep RESULT
python -m evidencedesk.docgen --check
python -m evidencedesk.gate | sed -n '4,9p;$p'

495 passed, 1 xfailed
GET /health -> 200 {"status": "alive"}
GET /ready -> 200 {"chunks": 36, "offline": true, "ready": true, "reason": ""}
POST /ask -> 200 answered req-1: Refunds go back to the original payment method within 5 business days after the warehouse receives the item. [kc-returns-v3:c2]
POST /ask -> 200 refused req-2: Not enough evidence in the help centre to answer this. Please contact Kestrel Cycles support.
POST /ask -> 401 refused req-3: I cannot help with that request. Please contact Kestrel Cycles support.
RESULT: ACCEPTED (0 failed, 1 known gap)
RESULT: READY (0 failed, 0 skipped)
docs are up to date
  recall_at_k             0.882
  citation_validity       1.0
  abstention_correct      0.773
  tool_pass               1.0
  format_pass             1.0
  pass_rate               0.848
GATE: GO

The test count grows whenever tests are added (the practice lab adds its own), so look for the word failed, not for a fixed number. The gate lines are the evaluation numbers the rubric points at.

Tool note (as of 2026-10-12): Python 3.12 is supported until 2028-10, Python 3.10 reached end of life on 2026-10-01, and current numpy needs 3.12 or newer, which is why the project pins 3.12. uvicorn 0.54.0 is listed as a current ASGI server; the kit does not depend on it and no real server was started here, because the API was always called in-process. Check the current documentation before you serve it.

Cheat sheet: the architecture on one page

One request from the caller to the reply, with the two places where trust stops. A prompt is not a guardrail: each boundary is guarded by code and tested.

 UNTRUSTED: the caller, the documents, retrieved text, the model's words, tool output

 caller --POST /ask--> api.py        settings, request model, /health, /ready
 (x-user-id)              |
                          v  safefail.safe_answer: identity, task, input size, one reply shape
 ======== boundary 1: from here on the input is treated as hostile ===============================
 corpus/ --chunking--> embeddings, index --> rag.Retriever     (bm25 is the keyword baseline)
                          |  evidence labelled untrusted, injection signals flagged, secrets screened
                          v
 context + budgets --> llm.LLMClient (fake model, or an adapter)    usage ledger, cache, limits
 ======== boundary 2: the model is not trusted either ============================================
 JSON reply --> results, contracts (stop reason, schema) --> citations (ids resolve, claim supported)
              --> tools + permissions (allowlist, call limits, approval) --> secret screen
                          |
 caller <-- {status, message, request_id}     no codes, no paths, no stack traces; log redacted

 Beside the request path, offline:  evals + gate (compare with the baseline), release + docgen (docs)

When something fails, find the arrow, then the module that guards it, then the test that pins it. Imports point down: api to safefail to security and on to llm.

Troubleshooting checklist

Each row is symptom, likely cause, the check that confirms it, and the fix. The seven failures after the table were reproduced in throw-away copies; the messages are real.

Each case breaks a fresh copy, so nothing you break stays broken.

fresh() { rm -rf ../scratch && cp -r . ../scratch && cd ../scratch; }   # every case breaks a throw-away copy

( fresh; echo "--- 1 corpus folder missing"
  EVIDENCEDESK_CORPUS_DIR=/nonexistent python - <<'EOF'
import os
from fastapi.testclient import TestClient
from evidencedesk.api import create_app, load_app_settings

with TestClient(create_app(load_app_settings(os.environ))) as http:
    for path in ("/health", "/ready"):
        response = http.get(path)
        print(path, response.status_code, response.json())
EOF
)
( fresh; echo "--- 2 a data file edited"
  echo '{"id": "x"}' >> data/eval_extra.jsonl
  python -c "from pathlib import Path; from evidencedesk.evals import load_dataset; load_dataset(Path('data/eval_manifest.json'), None)" 2>&1 | tail -1 )
( fresh; echo "--- 3 a dependency pin loosened"
  sed -i 's/^numpy==.*/numpy>=2.5/' requirements.txt
  python -m evidencedesk.release | grep FAIL
  python -m pytest tests/test_api.py -k exact_pins | grep ^FAILED )
( fresh; echo "--- 4 a secret-shaped line"
  echo 'api_key = "abcdefghijklmnop1234"' > notes.txt
  python -m evidencedesk.release | grep FAIL )
( fresh; echo "--- 5 a generated document edited by hand"
  echo "edit" >> docs/LIMITATIONS.md
  python -m evidencedesk.docgen --check )
( fresh; echo "--- 6 started from the wrong folder"
  cd .. && env -u PYTHONPATH python -m evidencedesk.api 2>&1 | tail -1 | sed 's/^.*python: //' )
( fresh; echo "--- 7 a gate threshold that breaks the baseline"
  python -m evidencedesk.gate --min-score 0.9 | grep -E "^GATE|block:" )

--- 1 corpus folder missing
/health 200 {'status': 'alive'}
/ready 503 {'ready': False, 'reason': 'FileNotFoundError', 'chunks': 0, 'offline': True}
--- 2 a data file edited
evidencedesk.evals.DatasetDrift: eval_extra.jsonl changed (manifest 72ed5a9aa64c773e, file 66039b58eaa0483b): bump the version, update the manifest
--- 3 a dependency pin loosened
[FAIL] exact pins                      not pinned exactly: ['numpy>=2.5']
FAILED tests/test_api.py::test_requirements_are_exact_pins_and_the_lock_contains_them
--- 4 a secret-shaped line
[FAIL] secrets and sensitive fixtures  1 finding(s): ['notes.txt:1 hard-coded secret']
--- 5 a generated document edited by hand
out of date: LIMITATIONS.md
--- 6 started from the wrong folder
Error while finding module specification for 'evidencedesk.api' (ModuleNotFoundError: No module named 'evidencedesk')
--- 7 a gate threshold that breaks the baseline
GATE: BLOCK
  block:  critical case rc09 failed its abstention check
  block:  citation_validity None is below the floor 0.9
  block:  pass_rate 0.485 is below the floor 0.8
  block:  citation_validity fell from 1.0 to None

Case 1 is why the API has two endpoints: alive (200) but not ready (503), and reason holds only the exception type because the text may hold a path. Case 2 stops you from evaluating data that changed unnoticed. Case 3 is one cause breaking two layers. Case 4 prints file and line, never the secret. In case 6 the folder is the cause, not the code. In case 7 the gate works as designed: a threshold that refuses almost everything breaks the baseline, exit code 1.

A citation can fail for three different reasons, and the checker keeps them apart because the fixes differ.

from pathlib import Path

from evidencedesk.citations import check_citations
from evidencedesk.evals import build_retriever

retriever = build_retriever(Path("corpus"))
question = "How long do I have to return a new bike?"
shown, known = retriever.retrieve(question), set(retriever.chunks)
outside = next(i for i in sorted(known) if i not in {r.chunk.id for r in shown})
good = "You can return a new bike within 30 days of delivery."
for label, claim, cited in (("invented id", good, ["kc-nope:c9"]), ("real id, not shown", good, [outside]),
                            ("real id, wrong number", good.replace("30", "45"), ["kc-returns-v3:c0"]), ("valid", good, ["kc-returns-v3:c0"])):
    r = check_citations(claim, cited, shown, known)
    print(f"{label:<22} ok={r.ok} invented={r.invented} outside={r.outside} missing_numbers={r.support.numbers_missing if r.support else None}")

invented id            ok=False invented=('kc-nope:c9',) outside=() missing_numbers=None
real id, not shown     ok=False invented=() outside=('kc-assembly-v1:c0',) missing_numbers=None
real id, wrong number  ok=False invented=() outside=() missing_numbers=('45',)
valid                  ok=True invented=() outside=() missing_numbers=()

Row one: the model invented an id. Row two: the id exists but was not in front of the model. Row three: a real, shown id, yet the claim says 45 days and the text says 30. A valid id is not support. Even row four only means the words and numbers appear in the passage, not that it answers the question.

What to review before the exams

Quiz 1 covers API and context basics (Parts 1 and 2), Quiz 2 covers building LLM features (Parts 3 to 6), and the Hard Exam covers all ten Parts with emphasis on the later ones. Each Hard Exam scenario theme has a place in the kit to practise.

Questions ask for judgement: given an error or an output, what is the cause and what do you change? Model names, prices and SDK signatures are not examined.

What next: the road to Advanced

Advanced extends this same project into KnowledgeOps. Your failures are the reasons for the first chapters.

What Beginner deliberately left out. Real provider calls (adapters are sketches), real embedding models, hybrid search and reranking, MCP, agents, fine-tuning, tracing, a real server, real authentication and anything cloud-specific. Training models and large infrastructure belong to other Level 1 paths, not to this one.

Honest limits of this kit. Say them in your own portfolio too:

• The model is a fake with recorded or rule-based replies. Nothing here measures the quality of a real model.
• The embedder is a toy that matches letters, not meaning. It explains the misses, and one test is a documented expected failure.
• The corpus is synthetic and small (13 documents, 36 chunks). The Bangla documents, questions and this Bangla tutorial were written by AI and have not been reviewed by a native-speaking editor; treat Bangla results as a smoke test, not as evidence.
• The evaluation set is small and has no held-out part. A service gap (rc15) showed that the measured pipeline was not the shipped one.
• It was run in-process on one operating system. No server, no concurrency, no load, no real authentication.

Security note (as of 2026-10-12): the UK NCSC wrote on 2025-12-08 that prompt injection may never be totally mitigated, so design must limit what a successful injection can do. OWASP publishes its LLM application risks in editions (the 2025 edition is confirmed; a 2026 edition was seen only in a secondary source): cite risks by name and edition, not by number. For Advanced: the MCP Python SDK has a 2.x release while 1.x is still released, and high-severity advisories of July 2026 were fixed on 1.x; whether 2.x carries the fixes is unconfirmed, so pin the version and read the advisories. Check the current documentation.

Where this shows up at work

Kwame in Accra takes over a prototype another team built. Before reading any code he runs the acceptance script, the release check and the gate, and reads the limitations file. In twenty minutes he knows what works, what is claimed but untested, and what to fix first. Teams call this a "definition of done" or a release readiness review; yours is a script, and it also serves an audit and your portfolio.

Common mistakes

• Reading ACCEPTED as "good". The checklist proves that controls exist and known cases behave. It does not prove quality on real users or a real model. Say so in the report.
• Hiding a known gap. Listing it with a reason is stronger than a green check. Make the list testable, so a fixed or a new gap fails the check.
• Measuring one pipeline and shipping another. Run the checks against the path users reach. Keep the evaluation and the service on the same code.
• Tuning on the failing case. A threshold chosen because it fixes one question is fitted to that question. Add cases first, then choose.
• Trusting totals. A hand-typed test count or score drifts. Read failures and generated numbers, and keep the manual release checklist (git history, licences) as well.

Try it yourself

1. Easy. Revisit Documentation, Release, and Portfolio Quality. In a copy of the project add a file with a hard-coded secret-shaped line. Predict which rows of the acceptance script fail and what the last line says, then run it.
2. Medium. Revisit Budgets, Caching, and Usage Limits. Give create_app a loader whose retriever has a minimum score of 0.36. Compare 0.3 and 0.36 over the 16 cases in data/rag_cases.jsonl: how many answerable questions are answered, and which unanswerable one is answered? Why would you not change the default yet?
3. Hard. Revisit Safe Failure and Security Regression Tests. Write tests/test_gap.py with two tests for rc15: a strict xfail that expects the default service to refuse it, and a test that passes with the stricter loader. What happens to the first test when someone fixes the default?

Answers

Easy. Only the release check sees the file, so one row fails (M5) and the result is NOT ACCEPTED; the known gap is still reported.

rm -rf ../scratch && cp -r . ../scratch && cd ../scratch
echo 'password = "abcdefghijklmnop1234"' > notes.txt
python -m capstone.accept | grep -E "FAIL|RESULT"

[FAIL] M5 evaluation, packaging and report             10 release checks, failing: ['secrets and sensitive fixtures']
RESULT: NOT ACCEPTED (1 failed, 1 known gap)

Medium. At both values 11 of the 13 answerable questions are answered (the same two misses, rc03 and rc06); only rc15 changes, and it is refused at 0.36. It looks free, but the value was found from one case. The generated docs/NEXT-STEP.md makes the same point: add more unanswerable questions (in Bangla too), repeat the sweep, then choose. Better still, make the service use the pipeline the evaluation measures.

from pathlib import Path

from fastapi.testclient import TestClient

from evidencedesk.api import create_app
from evidencedesk.chunking import chunk_corpus
from evidencedesk.citations import load_rag_cases
from evidencedesk.rag import Retriever

cases = load_rag_cases(Path("data/rag_cases.jsonl"))


def loader(min_score):
    return lambda settings: Retriever(chunk_corpus(settings.corpus_dir), clock=lambda: "2026-10-13T09:00:00Z", min_score=min_score)


for min_score in (0.3, 0.36):
    with TestClient(create_app(loader=loader(min_score))) as http:
        status = {c.id: http.post("/ask", json={"question": c.question}, headers={"x-user-id": "mateo"}).json()["status"] for c in cases}
    print(min_score, "answered:", sum(status[c.id] == "answered" for c in cases if c.answerable), "of", sum(c.answerable for c in cases),
          "| unanswerable but answered:", [c.id for c in cases if not c.answerable and status[c.id] != "refused"])

0.3 answered: 11 of 13 | unanswerable but answered: ['rc15']
0.36 answered: 11 of 13 | unanswerable but answered: []

Hard. A strict expected failure is a documented gap that the suite keeps honest.

# file: tests/test_gap.py
import pytest
from fastapi.testclient import TestClient

from evidencedesk.api import create_app
from evidencedesk.chunking import chunk_corpus
from evidencedesk.rag import Retriever

Q = "Do you sell electric scooters?"


def status(loader=None):
    kw = {"loader": loader} if loader else {}
    with TestClient(create_app(**kw)) as http:
        return http.post("/ask", json={"question": Q}, headers={"x-user-id": "mateo"}).json()["status"]


@pytest.mark.xfail(strict=True, reason="known gap: the service answers it (accept.GAPS lists it)")
def test_default_service_refuses_it():
    assert status() == "refused"


def test_a_stricter_threshold_refuses_it():
    strict = lambda s: Retriever(chunk_corpus(s.corpus_dir), clock=lambda: "2026-10-13T09:00:00Z", min_score=0.36)
    assert status(strict) == "refused"

python -m pytest tests/test_gap.py | tail -1 | sed 's/ in [0-9.]*s//'

1 passed, 1 xfailed

If someone fixes the default service, the first test passes unexpectedly and strict mode turns that into a failure. The person must then delete the marker and the GAPS entry together, so list and tests cannot disagree.

Pass checklist:

• Easy: you predicted M5 only, NOT ACCEPTED, 1 failed, and can say why A1 does not fail.
• Medium: the same count of answered answerable questions, rc15 no longer answered at 0.36, and a reason that mentions choosing a threshold from one case.
• Hard: the output ends with one pass and one expected failure, and you can explain strict mode.
• Whole capstone: 0 credentials needed, capstone.accept ACCEPTED with the gap listed, release check READY with the suite, docs current, and a limitations file that matches what you really ran.

Summary

• The capstone is the project you already built, proved by five milestone checks and a six-item acceptance checklist that runs as a script.
• Grade with evidence: each rubric category points at a test, a generated document or a release check, and says what it cannot show.
• One cheat sheet carries the commands, the request path with its two trust boundaries and a troubleshooting table made from failures you reproduced.
• The honest limits are part of the deliverable: a fake model, a toy embedder, a small synthetic corpus, Bangla written by AI, in-process runs only and a listed service gap.
• Advanced starts from your failures: better retrieval, protocols, agents, deeper evaluation, operations, security and deployment, on the same code base.

Next: AI Engineering with LLMs: Advanced takes this project to production-oriented retrieval, MCP, agents, evaluation, fine-tuning, LLMOps, security and deployment. You finished Beginner: keep the repository, the report and the limits list, because they are where Advanced begins.