Capstone, Cheat Sheet and What Next

Kwame in Accra is handing the KnowledgeOps release candidate to a reviewer who has never seen it. She will ask three things: what exactly was delivered, what do the numbers allow, and what happens when it breaks at night. A capstone is finished when she can answer all three from the repository in an hour, without him in the room. This page is that hour: the brief and its ten deliverables checked by a script, the rubric and the release rule, a cheat sheet, a runbook outline, a troubleshooting table made of failures reproduced for real, exam preparation and the road ahead. (An illustrative case.)

Everything runs offline. The model is a stand-in (real models are less predictable, which is exactly why you test), the clock is fake, and the honest result is still NO-GO: this page does not repair the release candidate, and the script it adds can never flip that verdict. It adds one folder, advcapstone/, and two test files, and edits nothing that exists. The graded exercises are on the page Practice Lab; they are not repeated here.

What you will learn

• The capstone brief, both options (RAG or controlled agent), the ten deliverables, and the earlier pages and kit modules behind each, with a check you can run.
• How the rubric maps to evidence, and what the release rule does when a high score meets a critical failure.
• A cheat sheet (verified commands, architecture patterns with trust boundaries, a runbook outline) and a troubleshooting table made of reproduced failures.
• How to prepare for the exams, and where to go next: specialised paths, what this course left out, and the honest limits of the kit.

The capstone brief and its ten deliverables

Goal. Deliver a release candidate for a narrowly scoped, realistic use case, and justify one architecture. The RAG option: permission-aware hybrid retrieval, reranking, citations, ingestion updates, measurable retrieval quality. The agent option: controlled planning, explicit state, tool permissions, bounded execution, human approval for consequential actions. Both need evaluation, tracing, security verification and deployment. Here the use case is the Kestrel staff help assistant and the choice made on Capstone Architecture and Integration was RAG. Nothing new is built: the capstone is assembling and proving.

The agent option changes rows 2 (task state, Agent Loops, State, and Memory), 5 (tool-use evaluation) and 8 (approvals and step budget, Human Approval and Multi-Agent Boundaries). The MCP pages (MCP Architecture and Local Server, MCP Security and Interoperability) apply to either option.

Run the acceptance script

The Beginner capstone (Capstone, Cheat Sheet and What Next) turned its checklist into capstone/accept.py. This one does the same for ten deliverables, with the same three statuses, and starts the same way: it runs the release candidate once and reads the evidence. ok: the artifact exists and its own checker is clean. gap: the artifact exists and the release evidence lists an open finding about it (the script assigns each of the 13 blockers of the verdict to the deliverable it belongs to). FAIL: the artifact is missing or contradicts the code. Like the Beginner list of known gaps, KNOWN_COPIES lets exactly one known problem through; a mismatch in either direction fails.

# excerpt of kit/advcapstone/accept.py
KNOWN_COPIES = {("p10", "g01")}


def d4(c: Ctx) -> Result:
    dev, hold = dataset.split(dataset.load_cases())
    copies = {(a, b) for a, b, _ in leakage.leakage_report(dev, hold, {})["dev_holdout"]}
    p = [] if hold else ["no holdout"]
    p += [f"unlisted near-copies {sorted(copies - KNOWN_COPIES)}"] if copies - KNOWN_COPIES else []
    p += [f"listed near-copy not found {sorted(KNOWN_COPIES - copies)}: remove it from KNOWN_COPIES"] if KNOWN_COPIES - copies else []
    note = f"{len(dev)} dev / {len(hold)} holdout, planted copy {'~'.join(sorted(copies)[0])}, {len(criteria.load_criteria())} criteria; the {len(gate_cases())}-case gate set has no holdout"
    status, text = finish(c, 4, p, note)
    return ("gap", text) if status == "ok" else (status, text)

The evaluation-design page planted a near-copy (p10 of g01) to test the leakage check; d4 expects exactly that pair. Run the list.

python -m advcapstone.accept

ADVANCED CAPSTONE ACCEPTANCE knowledgeops
[ok  ] D1  architecture, trust boundaries   7 flows, 6 boundaries, every control imports; threat gaps 4 -> 0
[ok  ] D2  versioned ingestion              snapshot v3, 12 sources versioned, verify findings 0, regression go
[gap ] D3  provider-neutral, offline fake   2 tool contracts, no provider import, offline stand-in by default
        open in the release evidence: M5
[gap ] D4  dataset, holdout, metrics        20 dev / 11 holdout, planted copy p10~g01, 19 criteria; the 19-case gate set has no holdout
        open in the release evidence: F1, F4, K1, K3, M2
[ok  ] D5  retrieval benchmark              7 variants, 24 questions, best mrr 0.897 (hybrid+overlap)
[ok  ] D6  judge experiment, calibration    30 labelled pairs, agreement 0.633, kappa 0.456
[gap ] D7  logs, traces, latency, cost      19 traces well formed, p95 0.31 s (fake clock), 13 model calls
        open in the release evidence: M3
[gap ] D8  threats, isolation, risks        123 isolation attempts held, probes 30/31, 8 release risks
        open in the release evidence: probe e:2, K6, M1, M6
[gap ] D9  deployment, health, rollback     17 smoke steps, runbook and rollback consistent, no container run
        open in the release evidence: M4, M7
[ok  ] D10 final release decision           NO-GO, 13 blockers, each with a fix
RUBRIC (weights add to 100)
   15%  Architecture and implementation quality    architecture.problems, D1 D3
   15%  Retrieval or agent behavior                retrieval.benchmark, D5
   15%  Evaluation quality and regression gates    evalkit.ci, gate, D4 D6
   20%  Security and data isolation                secgate, isolation, probes, D8
   10%  Observability, cost, and latency analysis  REPORT.md, traces, D7
   10%  Deployment, resilience, and recovery       serve.readiness, drill, D9
   10%  Reproducibility and documentation          repro, docgen --check, D10
    5%  Trade-off analysis and portfolio defence   design review
RELEASE RULE: a high total cannot compensate for a critical failure
  [ok  ] cross-tenant leakage               123 attempts, failures none
  [ok  ] unauthorised actions               31 probes, harm markers seen none
  [ok  ] exposed credentials                secret scan findings 0
  [skip] non-reproducible mandatory tests   repro drift 0; suite: not run (pass --tests)
RELEASE DECISION (from knowledgeops.verdict, not changed here): NO-GO (13 blockers)
RESULT: DELIVERABLES COMPLETE (0 failed, 5 with open findings); release decision NO-GO

Each gap line is a real finding. D3: nobody has read the licence and data-rights findings (M5). D4: answer floors F1 and F4 fail (K1), no real-model run exists (K3, M2), and the 19-case gate set has no holdout of its own (the holdout is in the 31-case evaluation set); the last point is new on this page, the verdict does not list it. D7: no load test with a real model (M3). D8: probe e:2, a question the documents cannot answer (“Do you deliver bikes to the moon?”), is answered; provider terms (K6, M1) and risk owners (M6) are unsigned. D9: no container was built (M4); token handling is unconfirmed (M7). Deliverables complete and release NO-GO are different questions.

The script cannot show that answers are good (the model is a stand-in, 19 cases are few) or that a person reviewed anything. It guards against forgetting; judgement stays with a person.

Choose RAG or an agent, and defend it

Reuse the decision record of Capstone Architecture and Integration instead of arguing from taste. It derives the choice from what the use case needs: an agent only when a need cannot be met without one (an action that changes something outside the system, steps whose number and order are not known in advance, or open-ended tool choice). The first run uses the use case as written; the second flips one need.

from evidencedesk.knowledgeops import decision, usecase

uc = usecase.load_usecase()
for flag in (None, "write_actions"):
    needs = uc.needs.model_copy(update={flag: True}) if flag else uc.needs
    r = decision.decide_architecture(uc.model_copy(update={"needs": needs}))
    print(f"{flag or 'as written':<14}-> {r.decision:<6} triggers {r.triggers} problems {r.problems()}")
r.triggers = []
print("agent with no trigger ->", r.problems())

as written    -> rag    triggers [] problems []
write_actions -> agent  triggers ['write_actions'] problems []
agent with no trigger -> ['agent chosen but no requirement needs it: choose the simpler architecture']

Defend the choice by naming what you give up. RAG keeps a fixed, testable pipeline whose permission bug can only live in retrieval, and gives up questions that need two lookups in sequence. An agent adds that flexibility and costs a step budget, state that survives a restart, approvals a tired person may click through, and many more failure traces to evaluate. The design review (Release Evidence and Production Readiness) records the alternatives, the trade-off and the evidence test.

The grading rubric and its evidence

The capstone is graded out of 100 in eight categories. A grade without evidence is an opinion, so each row names where the evidence is and what that evidence cannot show.

The weights add to 100, and security and isolation carry the most (20). The model is not a category: the course grades the engineering around it. The script prints the weights and checks they add to 100; it invents no score for KnowledgeOps, because a score needs a reviewer.

The release rule, with a worked example

A high total cannot compensate for a critical security failure. Cross-tenant leakage, unauthorised actions, exposed credentials or non-reproducible mandatory tests require remediation before passing. accept.rule(scores, failed) computes the weighted total (score per category from 0 to 1 times its weight) and then lets any critical failure override it.

The example applies it to a hypothetical submission that scores 0.95 everywhere, with and without a tenant leak, then measures the four conditions on the real candidate.

from advcapstone import accept

strong = {k: 0.95 for k in accept.WEIGHTS}   # a hypothetical submission that is excellent everywhere
print("no critical failure :", accept.rule(strong))
print("one tenant leak     :", accept.rule(strong, ["cross-tenant leakage"]))
ctx = accept.build_ctx()                      # the real KnowledgeOps release candidate
for name, ok, detail in accept.critical_gates(ctx, None):
    print(f"{name:<34}{'not measured' if ok is None else 'held' if ok else 'FAILED':<13}{detail}")

no critical failure : (95.0, 'PASS')
one tenant leak     : (95.0, 'REMEDIATE: cross-tenant leakage')
cross-tenant leakage              held         123 attempts, failures none
unauthorised actions              held         31 probes, harm markers seen none
exposed credentials               held         secret scan findings 0
non-reproducible mandatory tests  not measured repro drift 0; suite: not run (pass --tests)

The same 95.0 passes in one line and must be remediated in the next: the total did not change, the leak did. On the real candidate three conditions held (123 isolation attempts through three doors, 31 probes without a harm marker, no secret-shaped string); the fourth is not measured until you pass --tests, because a skipped suite is not a passed one. KnowledgeOps is NO-GO for other reasons (failing floors, unsigned sign-offs): a critical failure means “cannot pass whatever the score”, while the kit's verdict is stricter and refuses on any blocker.

Cheat sheet: the commands

Run everything from the kit folder (the one with pyproject.toml). Exit code 1 is often a decision, not a crash: a gate that says BLOCK and a verdict that says NO-GO exit 1 on purpose, so read the printed last line as well as the code. Setup (environment and pinned requirements) is the only step that needs the network and is not repeated.

python -m advcapstone.cheat --run runs them for real and compares each exit code and one expected output line. The evaluation gate exits 1 twice on purpose: the shipped candidate blocks, and even the baseline settings block, because a baseline is a record, not an approval.

python -m advcapstone.cheat --run

[ok  ] capstone checklist                   exit 0 (expected 0)
[ok  ] release candidate run                exit 1 (expected 1)
[ok  ] release candidate gate               exit 1 (expected 1)
[ok  ] release pack and decision            exit 1 (expected 1)
[ok  ] reproducibility record               exit 0 (expected 0)
[ok  ] evaluation gate (candidate v2)       exit 1 (expected 1)
[ok  ] evaluation gate (baseline settings)  exit 1 (expected 1)
[ok  ] security regression gate             exit 0 (expected 0)
[ok  ] security verification pack           exit 1 (expected 1)
[ok  ] threat model consistency             exit 0 (expected 0)
[ok  ] retrieval benchmark                  exit 0 (expected 0)
[ok  ] deployment readiness                 exit 0 (expected 0)
[ok  ] container files (static)             exit 0 (expected 0)
[ok  ] incident lab                         exit 0 (expected 0)
[ok  ] ingestion operations lab             exit 0 (expected 0)
[ok  ] release checks                       exit 0 (expected 0)
[ok  ] generated documents                  exit 0 (expected 0)

Tool note (as of 2026-10-12): the kit pins the MCP Python SDK at 2.3.0 and the protocol revision 2026-07-28 (VERIFY: both change often, and the SDK has had several security advisories, so re-check the pins at every release). Check the current documentation before you change them.

Cheat sheet: the architecture on one page

Eight patterns, each with the path a request takes and where trust stops. A prompt is not a guardrail: each boundary is guarded by code and has a test. B1 to B7 are the threat model's boundary ids.

1 RAG                caller --B1--> edge --> pipeline --B5--> index (tenant + ACL filter BEFORE ranking)
                     pipeline --B2--> model --B3--> checks (schema, links, secrets) --> reply, cited chunk ids
2 Hybrid + rerank    query --> sparse + dense --> fusion --> reranker (top N only) --> top k to the context
                     the filter runs first; scores are signals, not probabilities
3 MCP boundary       app --B6--> tool server over stdio; tool list pinned, arguments validated, results untrusted
                     the protocol gives interoperability, not authentication
4 Controlled agent   plan --> step budget --> tool gate (allow-list, contract) --B4--> read-only tools
                     consequential action --> approval queue --> a person decides --> checkpoint, resume
5 Multi-agent        coordinator --> planner / researcher / reviewer; one budget; typed envelopes;
                     a worker's text is data for the next one, never an instruction
6 Ingestion          source --> parse --> clean --> classify --> quarantine hostile --> chunk --> index
                     versioned manifests, stable ids, ACL stored per chunk, verify() after every run
7 Observe            request --> spans per stage --> events with lengths, not content --B7--> redaction
                     metrics, SLOs, alerts, budgets; incident --> runbook --> blameless postmortem
8 Deployment         edge (token, size, quota, admission) --> app --> /health, /ready with reason codes
                     secrets as files, pinned lock file, canary behind a gate, a rollback plan per unit

B1 user>app  B2 app>model  B3 model>app  B4 app>tools  B5 app>stores  B6 app>third parties  B7 app>observers
UNTRUSTED: the caller, documents, model replies, tool results, other agents' text. Trusted only after code checks it.

Operational runbook outline

Someone tired reads a runbook at night, so it is short, ordered and owned by roles, not names. Reuse the outline of Release Evidence and Production Readiness (knowledgeops.runbook): routine tasks with an owner; one entry per incident category (severity, owner role, trigger, ordered steps, verification); a first move (kill switch or search-only mode, no deploy needed); a rollback plan, one row per release unit (application, dependency, model, prompt, config) with its previous version, the check and the minutes; drills on a fake clock; and a blameless postmortem (Monitoring, Budgets, and Incidents). The code checks the outline: every category has an entry, the leak entry starts by stopping answers, no person is named, every rollback unit points at an existing test.

from evidencedesk.knowledgeops import runbook

book = runbook.load_runbook()
for name, e in book["incidents"].items():
    print(f"{name:<11}{e['severity']:<6}{e['owner']:<18}{len(e['steps'])} steps")
print("runbook problems:", runbook.problems())
for d in runbook.drill():
    print(f"{d.step:<30}{d.code:<32}breaker {d.state}")

leak       SEV1  security on-call  5 steps
security   SEV2  security on-call  4 steps
provider   SEV2  app on-call       4 steps
retrieval  SEV3  data on-call      4 steps
regression SEV3  app on-call       3 steps
overload   SEV3  app on-call       3 steps
runbook problems: []
provider failing, request 1   failed/server                   breaker closed
provider failing, request 2   failed/server                   breaker closed
provider failing, request 3   failed/server                   breaker open
breaker open: search only     answered/search_only            breaker open
31 s later: one trial call    answered/ok                     breaker closed
kill switch pulled            failed/temporarily_unavailable  breaker closed

The drill: three provider failures open the breaker, the next request gets search-only results with no model call, a trial call 31 seconds later closes it, and the kill switch stops answers. It proves the mechanism, not the minutes in the plan.

Troubleshooting: symptom, cause, check, fix

Every row was reproduced: python -m advcapstone.cheat --trouble breaks one thing in a throw-away copy of the kit (or builds bait on demand) and prints the evidence.

python -m advcapstone.cheat --trouble

DatasetDrift: file changed
    evidencedesk.evals.DatasetDrift: rag_cases.jsonl changed (manifest <hash>, file <hash>): bump the version, update the manifest
ci says 'changed: ...' and the numbers are not comparable
    review baseline: not comparable: dataset differs (<hash> vs <hash>)
/ready answers 503
    GET /ready -> 503 {"chunks": 0, "offline": true, "ready": false, "reason": "empty_index"}
the gate says BLOCK on a floor
    floor: answer_correct 0.667 (8/12) is below the floor 0.85
the secret scan reports a finding
    notes.md:1 provider-style key
an old answer cites a passage that is gone
    kc-stores:c0   gone    This passage no longer exists.
the MCP server and client disagree
    sdk 2.2.0 != pinned 2.3.0
repro says datasets changed
    datasets.knowledgeops/cases.jsonl: <hash> -> <hash>

Hashes are masked as <hash>. Two states are not faults: secpack says NOT READY by design (finding F-105 is open, one training sign-off is pending), and knowledgeops says NOT READY because the floors fail.

Prepare for the exams

Each tutorial has 35 questions: Quiz 1 (10), Quiz 2 (10) and the Hard Exam (15). They test application and judgement, not model names or prices: code-reading and output questions use short code with stated inputs, debugging questions give an error or a failing assertion, design questions want a rationale and a trade-off. None needs a key, a GPU or a paid service. The kit tests are the best practice, because the questions are built the same way.

for group in "retrieval mcp_local" "agent evalkit robust finetune" "observe threat guard deploy scale ingest retops knowledgeops readiness"; do
  python -m pytest --collect-only $(for g in $group; do echo tests/test_${g}_*.py; done) | tail -1 | sed "s/ in [0-9.]*s//"
done

157 tests collected
274 tests collected
715 tests collected

The three lines are the three groups above (for the Hard Exam, its emphasis). Its scenario themes map to tests you can run and read:

A method that works: open a test, cover the assertion, predict the result, then run it; when wrong, find the page that taught the rule. Questions about an SDK's current behaviour are marked VERIFY with a pinned version: learn the habit (pin, then check), not the signature.

What next

You can now explain each component, run the code, test expected and failure behaviour, and justify the design: the standard of this course, which never needed a paid model or a GPU. It stopped on purpose where specialist depth begins. Pick the path that matches the work you want:

What this course did not cover is that table: it taught the engineering around a model, with application-level threat modelling, not a security or governance qualification, legal advice, a platform or a business launch. The honest limits of the whole kit belong in your limits file too: the embedder is a toy, so retrieval numbers say little about real embeddings; every model is a fake or a stand-in, so there is no measurement with a real language model; load and latency are simulated on a fake clock; the Bangla pages and cases were written by AI and have not been reviewed by a native-speaking engineer; nothing ran in a container, because Docker was not run; and 19 to 31 cases per set are small samples with wide intervals. That is why every result above is offline evidence only.

Where this shows up at work

Every team that ships an LLM feature meets the same meeting: a reviewer, a date and evidence of uneven quality. This page's skills decide it: a deliverable list a script checks, a rubric with evidence, a release rule written before the numbers, a runbook someone else can follow and an honest limits list. A NO-GO with a path to GO earns more trust than a GO without one.

Common mistakes

• Grading by the total. A 95 hides a leak; check the critical conditions first.
• Lowering a floor to turn a gate green. The Hard exercise shows how it gets caught. Change a threshold only with new evidence and a recorded reason.
• Calling a deliverable done because a file exists. Run its checker; keep open findings visible as gaps.
• Misreading exit code 1. Know which commands exit 1 on purpose; read the last line.
• A troubleshooting table nobody reproduced. It goes stale in a month; generate the evidence.

Try it yourself

1. Easy. Predict which line of python -m advcapstone.accept --tests differs from the run above and what the last line says, then run it (about a minute; the passed count is masked as N because it grows). Revisit Packaging, Configuration, and Reproducibility: why does the suite run with every key variable removed?
2. Medium. Deliverable 10 allows a conditional go. Write outcome(total, critical, floors_failed, only_accepted_open) so a critical failure or a failing floor never becomes a conditional go; apply it to today's KnowledgeOps, to a future where the floors pass, and to a 98-point submission with a tenant leak. Revisit Robust Metrics and Release Gates for why a floor is not a trade.
3. Hard. Turn the gate green the wrong way: in a throw-away copy of the kit, lower floors F1 to 0.66 and F4 to 0.8, then run the gate and the release command. What catches it, and what is the decision? Revisit Release Evidence and Production Readiness for how the verdict treats a failing criterion.

Answers

Easy. Only the last critical condition changes, from not measured to held; the last line is the same, because deliverables and verdict do not depend on the suite.

python -m advcapstone.accept --tests | grep -E 'non-reproducible|^RESULT' | sed -E 's/[0-9]+ passed/N passed/'

  [ok  ] non-reproducible mandatory tests   repro drift 0; suite: ok N passed, 1 xfailed
RESULT: DELIVERABLES COMPLETE (0 failed, 5 with open findings); release decision NO-GO

Medium.

from advcapstone import accept

def outcome(total, critical, floors_failed, only_accepted_open):
    if critical or floors_failed:
        return "no-go"
    return "conditional go" if only_accepted_open else "go"

ctx = accept.build_ctx()
floors = [v.id for v in ctx.ev.failing]
print("today        :", outcome(82.0, [], floors, False), floors)
print("floors fixed :", outcome(82.0, [], [], True))
print("a tenant leak:", outcome(98.0, ["cross-tenant leakage"], [], True))

today        : no-go ['F1', 'F4']
floors fixed : conditional go
a tenant leak: no-go

Hard.

import json
from pathlib import Path
from advcapstone.cheat import copy_run

def lower_floors(work: Path) -> None:
    path = work / "data/knowledgeops/acceptance.json"
    data = json.loads(path.read_text(encoding="utf-8"))
    for c in data["criteria"]:
        c["value"] = {"F1": 0.66, "F4": 0.8}.get(c["id"], c["value"])
    path.write_text(json.dumps(data, indent=1), encoding="utf-8")

for module in ("evidencedesk.knowledgeops.gate", "evidencedesk.knowledgeops.release --out pack"):
    out = copy_run(lower_floors, module)
    print("\n".join(l for l in out.splitlines() if l.startswith(("GATE", "criteria", "DECISION", "  docs", "  risk"))))

GATE: GO
criteria met 19/19; gate go; probes 30/31; smoke 17/17
DECISION: NO-GO (12 blockers)
  docs       1  repro
  risk       3  K1, K3, K6

The gate says GO and 19 of 19 criteria pass, yet the decision stays NO-GO with 12 blockers: the reproducibility record flags that acceptance.json no longer matches the committed record, and K1, K3 and K6 remain, because the model, the measurement and the provider data flow did not change. Lowering the floors removed two blockers and added one.

Pass checklist. Easy: one line changed, not measured to held, same final result. Medium: no-go, conditional go, no-go, with critical and floors checked first. Hard: gate GO, 19/19 criteria, a repro blocker, NO-GO with 12 blockers, and you can say why a floor changes only with new evidence. Whole capstone: 0 failed, every gap line names a finding you can explain, the verdict is the one the evidence supports, docs are current, and your limits file lists what you did not measure.

Last, run this page's own tests:

python -m pytest tests/test_advcapstone_accept.py tests/test_advcapstone_cheat.py | tail -1 | sed 's/ in [0-9.]*s//'

25 passed

Summary

• The capstone is the system you built, proved by ten deliverable checks that report ok, gap or FAIL honestly; the candidate is NO-GO and the script never changes that.
• Choose RAG or an agent from the needs with the decision record, and name what you give up.
• Grade with evidence per category; a critical failure overrides any total, and a floor is never lowered to fit a result.
• One cheat sheet: verified commands with real exit codes, eight architecture patterns with trust boundaries, a runbook outline and a table of reproduced failures.
• The honest limits are part of the deliverable: a toy embedder, fake models, simulated load, AI-written Bangla, no real-model measurement, no container run.

Next: there is no further page. You have finished AI Engineering with LLMs: Advanced. Keep the repository, the release pack, the limits list and your own notes, because they are your evidence. Then take the exams, and choose a specialised path: ML & Data, AI Infra & Ops, Security & Governance, Business & Product, or Automation & Freelancing, each of which starts where this course deliberately stopped.