Practice Lab

Priya in Pune is asked to prove that she can build an AI feature that a team can trust. A certificate does not show that. A graded run does: a fixed set of cases, a score, and a list of things that fail the whole job however many points you collect. This page is that run for the whole tutorial. You build one small knowledge assistant in six challenges, and a grader scores it out of 100.

Everything is offline. The model is a stand-in, not a language model; real models are less predictable, which is exactly why you test. The grader needs no key, no network and no download. This page adds a folder kit/lab/ and the test file kit/tests/test_lab.py, and changes nothing that exists.

What you will learn

• How the lab is built: six challenges, 100 points, a pass mark of 70, and four automatic critical failures that outrank the score.
• How to turn each earlier Part into a fixture and an automated pass criterion, with the starter files, hints and a model solution.
• How to prove that a grader catches what it claims to catch, using a passing reference and deliberately failing submissions.
• What a fixture grader cannot show, and how to submit your work with a README, example output and a design note.

How the lab works

The lab is called "Build a Reliable Knowledge Assistant". You write six small Python modules in a folder called submission/. A grader imports them, feeds them recorded replies and a fixed corpus (the Kestrel Cycles help centre again), and adds up points. You may import the whole evidencedesk package: it is your own project, and good engineers reuse their tested parts.

Pass: at least 70 points, and no critical failure, and a complete submission (list below). Four automatic critical failures end the attempt whatever the score: a committed credential, model-generated text executed as code, a tool authorization bypassed, and a fabricated citation returned as if it were verified. A missing key or connection never blocks a pass: the grader switches the network and key variables off while it runs.

Required submission (Part 10): source code, a fixture dataset of your own, tests, a README, example output and a short explanation of your design trade-offs. The grader checks that these exist, are not placeholders (no TODO, a minimum length), and that your own tests pass. It cannot judge whether they are good.

Start from the starter. Every stub raises NotImplementedError and its docstring is the contract: names, argument types, return shapes and codes.

cp -r lab/starter submission
ls submission

client.py
DESIGN.md
example_output.txt
fixtures
rag.py
README.md
report.py
search.py
stream.py
tests
tools.py

# excerpt of kit/lab/starter/client.py
"""Challenge 1 (15 points): one internal contract behind a fake client and two wire formats.

Read fixtures: lab/fixtures/wire.json (request and reply shape of wire A and wire B, and two error replies).
"""


def make_client(kind: str, transport=None):
    """kind is 'fake', 'wire_a' or 'wire_b'.

    - 'fake': an object with complete(request) -> LLMResponse that answers from recorded text and raises LookupError for an unknown question.
    - 'wire_a' / 'wire_b': an adapter. transport(payload_dict) -> reply_dict stands in for the network. Map our LLMRequest to the wire
      payload, map the reply back to an LLMResponse (text, usage, stop_reason in our words: end, length, other), and turn a non-200
      status into evidencedesk.errors.from_status(...). Anything else: raise ValueError.
    """
    raise NotImplementedError("TODO challenge 1")

Run the grader on the untouched starter. The score is 0, and that is the right answer. Notice what else it shows: no critical check fails, because nothing ran that could do harm, and the submission list names exactly what is still missing.

python -m lab.grade submission

1. Provider-neutral client         0/15
2. Structured streaming            0/15
3. Local embeddings and search     0/15
4. Citation-based RAG              0/25
5. Safe tool calling               0/15
6. Evaluation and cost controls    0/15
Critical checks
  [ok  ] no committed credentials
  [ok  ] no model-run code
  [ok  ] tool authorization kept
  [ok  ] no fake citation verified
Required submission
  [FAIL] README.md              needs 300+ characters and no TODO
  [FAIL] DESIGN.md              needs 300+ characters and no TODO
  [FAIL] example_output.txt     needs 50+ characters and no TODO
  [FAIL] fixture dataset        fixtures/ needs your own data file (no TODO)
  [FAIL] tests written          0 test function(s), need 3+
SCORE 0/100 | critical failures 0 | RESULT: FAIL

Fixtures and the grader

Fixtures are public files in lab/fixtures/: JSON lines, one case per row. Read them first; they are the specification. Two rows show the idea: a recorded stream with the status it must produce, and a recorded model reply that cites an id that exists nowhere.

import json

for name, n in (("stream_cases", 0), ("rag_cases", 5)):
    row = json.loads(open(f"lab/fixtures/{name}.jsonl", encoding="utf-8").read().splitlines()[n])
    print(name)
    for key, value in row.items():
        print(f"  {key}: {value}")

stream_cases
  id: s1
  check: assemble
  events: [['chunk', 'Standard deliv'], ['chunk', 'ery takes 4 '], ['chunk', 'to 7 days.'], ['done', '']]
  status: complete
  text: Standard delivery takes 4 to 7 days.
rag_cases
  id: f1
  check: fabricated
  question: How should I store the e-bike battery for a long time?
  reply: {'answer': 'Keep it at 50 percent in a dry place.', 'cited_ids': ['kc-battery-v9#s2'], 'confidence': 'high', 'status': 'answered'}
  expect: abstained

The grader (python -m lab.grade <folder> [--verbose] [--no-tests]) is one file. Each check is a small function registered with its challenge, name and points, and it fails by an AssertionError with a message. A broken file costs only its own challenge. An excerpt:

# excerpt of kit/lab/grade.py
def check(challenge: int, name: str, points: int):
    def register(fn):
        CHECKS.append((challenge, name, points, fn))
        return fn
    return register


def stream_check(group: str, ctx: Ctx):
    for row in (r for r in rows("stream_cases") if r["check"] == group):
        events = [StreamEvent(kind, text, stop_reason="end" if kind == "done" else None) for kind, text in row["events"]]
        status, text = ctx.m("stream").collect(events)
        assert (status, text) == (row["status"], row["text"]), f"{row['id']}: got {(status, text[:20])}"


for _group, _points in (("assemble", 4), ("interrupted", 3), ("error", 2), ("empty", 1)):
    check(2, f"stream_{_group}", _points)(lambda ctx, g=_group: stream_check(g, ctx))


def critical_checks(ctx: Ctx) -> list[tuple[str, bool, str]]:
    out = []
    hits = scan_secrets(ctx.folder, {})
    out.append(("no committed credentials", not hits, f"{len(hits)} finding(s): {hits[:2]}" if hits else ""))

Two honest limits. First, the grader checks fixtures only: a solution can pass by handling exactly these cases. Treat the fixtures as examples of a behaviour, then test more cases yourself. Second, points are all or nothing per check, so read the failure message before you change code.

Challenge 1: Provider-neutral client (15 points)

Task. Write make_client(kind, transport=None). Kind fake returns a client that answers from recorded text. Kinds wire_a and wire_b return adapters for two imaginary providers with different request and reply shapes. All three expose complete(request) -> response with our own types. Fixture: wire.json lists both wire formats and two error replies; the transport is a recorded function, so no network is used.

Hints. Provider-Neutral LLM Clients (the types and the fake); Adapters, Configuration, and Error Handling (status to error class). A good solution keeps provider words inside the adapter: the rest of the project never sees finish or done_reason. It never retries inside the adapter and never prints a response body that could hold internals.

Challenge 2: Structured streaming (15 points)

Task. In stream.py write collect(events) returning (status, text) and read_reply(response) returning (ok, code). Fixtures: stream_cases.jsonl (six recorded streams, one in Bangla, split mid-word) and reply_cases.jsonl (five model replies).

Hints. Streaming and Structured Results; Reliable Output Contracts. A good solution checks the stop reason before it parses, since a cut-off reply can still look like JSON, and treats valid JSON as only the first of three checks (syntax, schema, business rules).

Challenge 3: Local embeddings and search (15 points)

Task. In search.py write Search(corpus_dir) with search(query, k=3, min_score=0.3) returning [(passage_id, score)]. Fixture: search_cases.jsonl (eight queries) over the real corpus, which contains a superseded returns policy and Bangla pages.

Hints. Embeddings and Similarity; Local Indexes and Retrieval Quality. A good solution filters by status before ranking, and treats "nothing good enough" as an empty list the caller can act on. Remember that a high similarity score is a signal, not proof that a passage answers the question; the floor of 0.3 is a starting guess for this toy embedder, not a tuned value.

Challenge 4: Citation-based RAG (25 points)

Task. In rag.py write Assistant(corpus_dir).ask(client, question). It returns a dict with status (answered or abstained), text, citations, verified, reason and retrieved. An answer is verified only if every cited id was retrieved, exists, and the cited text contains the claim's numbers and key words. Anything else is the defined abstention: the text of the project's no-evidence message, no citations, verified false, and the model's own words never shown. Fixture: rag_cases.jsonl, eight questions, each with the recorded reply of the stand-in model.

Hints. RAG Ingestion and Citation Flow (retrieve, ask, cite); RAG Reliability and Evaluation Data (weak retrieval, citation support); Context Assembly and Trust Boundaries (retrieved text is data). A good solution builds citations only from what it retrieved, never from the model's list; it abstains before calling the model when retrieval is weak; and it knows that a valid id is not support. The support check is words and numbers, not meaning, so a clever false claim built from words in the passage would still pass. It also returns the same abstention for every reason, so a caller cannot learn internals; the specific reason goes in reason for the log.

Challenge 5: Safe tool calling (15 points)

Task. In tools.py write Tools(corpus_dir).run(raw, task, user, allowed_tasks) returning {ok, code, data}. Three tools are registered: calculator and lookup_document (read) and send_email_draft (write; the mail system is a list in self.outbox). The tasks are math, faq and reply; each may use one tool, and no write tool is ever granted in this lab. Fixture: tool_cases.jsonl, nine calls with the expected result and the accepted error codes.

Hints. Tool Contracts and Execution; Tool Permissions and Failure Handling; Prompt Injection and Access Boundaries (identity before tools). A good solution puts every rule in code outside the model, since a prompt is not a guardrail, answers "does not exist" and "not for this task" alike, and walks the syntax tree for arithmetic instead of calling eval.

Challenge 6: Evaluation and cost controls (15 points)

Task. In report.py write three things. run_eval(cases, ask) returns {total, passed, failures} and counts a crashing case as a failure. Guarded(inner, max_calls) wraps a client with an exact-match cache and a call budget: a repeat is free, a new request after the budget raises an error whose code is budget_exceeded. report_md(result, limitations) writes the counts, every failing id and the limitations, and refuses an empty list. Fixture: eval_set.jsonl, eight cases with a stub that gets three of them wrong on purpose (one answers a question it should refuse, one refuses a question it should answer, one crashes).

Hints. Evaluation Datasets and Deterministic Tests; Rubrics, Regression, and Release Gates (the report lists failures); Cost and Latency Fundamentals; Budgets, Caching, and Usage Limits (cache key, hard stop). A good solution checks the cache before the budget, never caches a failure, and writes the limitations first: the grader's own stub model, eight cases and one corpus.

Critical failures

These four checks are not worth points; they decide whether the points count. Each is an automated check, not a promise.

A grader you have not tried to fool is only an opinion. lab/bad.py builds a copy of the reference solution with exactly one flaw each. The flaw for the authorization check is typical: a "convenient" shortcut that sends the email the model asked for.

# excerpt of kit/lab/bad.py
    "auth": ("tools.py", '''

_gated_run = Tools.run


def _run_open(self, raw, task, user, allowed_tasks):  # "convenient": the model asked for an email, so send it
    if isinstance(raw, dict) and raw.get("name") == "send_email_draft":
        self.outbox.sent.append(dict(raw.get("arguments", {})))
        return {"ok": True, "code": "ok", "data": {"queued": True}}
    return _gated_run(self, raw, task, user, allowed_tasks)


Tools.run = _run_open
'''),

Grade the four copies. The key in the first copy is assembled at run time, so the lab's own files hold no key-shaped text. The folder bad/ is deleted afterwards because the release check of Documentation, Release, and Portfolio Quality would rightly refuse it.

for flaw in secret exec auth fabricated; do
  python -m lab.bad $flaw bad/$flaw > /dev/null
  echo "== $flaw"
  python -m lab.grade bad/$flaw --no-tests | grep -E 'FAIL|SCORE'
done
rm -rf bad

== secret
  [FAIL] no committed credentials 2 finding(s): ['local_settings.py:1 provider-style key', 'local_settings.py:1 hard-coded secret']
SCORE 100/100 | critical failures 1 | RESULT: FAIL
== exec
  [FAIL] no model-run code      tools.py:32 eval(); model text ran as code
SCORE 94/100 | critical failures 1 | RESULT: FAIL
== auth
  [FAIL] tool authorization kept t5 ran; t6 ran
SCORE 93/100 | critical failures 1 | RESULT: FAIL
== fabricated
  [FAIL] no fake citation verified f1 shown as an answer; o1 shown as an answer
SCORE 89/100 | critical failures 1 | RESULT: FAIL

Each copy fails exactly one critical check, and the grader names the file or case. Notice the scores: all four are 89 or more. Points alone would have passed all four, which is why the critical checks come first. See Safe Failure and Security Regression Tests for the habit behind them.

The reference solution

lab/reference/ answers all six challenges with thin layers over the project modules, plus a README, a design note, its own tests, a fixture file of its own cases and an example output that a test regenerates and compares. Try the challenges before you read it. Here is its real grade.

python -m lab.grade lab/reference

1. Provider-neutral client        15/15
2. Structured streaming           15/15
3. Local embeddings and search    15/15
4. Citation-based RAG             25/25
5. Safe tool calling              15/15
6. Evaluation and cost controls   15/15
Critical checks
  [ok  ] no committed credentials
  [ok  ] no model-run code
  [ok  ] tool authorization kept
  [ok  ] no fake citation verified
Required submission
  [ok  ] README.md
  [ok  ] DESIGN.md
  [ok  ] example_output.txt
  [ok  ] fixture dataset
  [ok  ] tests written          4 test function(s), need 3+
  [ok  ] own tests pass
SCORE 100/100 | critical failures 0 | RESULT: PASS

It scores 100 of 100 and passes with no critical failure and a complete submission. Read that number carefully. It was written knowing the fixtures, so it shows that the grader and the fixtures agree with a correct solution. It says nothing about questions outside the fixtures: the toy embedder misses paraphrases such as "How do I get my money back?", as the evaluation of RAG Reliability and Evaluation Data already showed. Your own cases in fixtures/ are where you look for such misses.

Optional real-provider extensions

Two optional extensions use a real model. They have no output here because none was produced, they are never graded, and a missing key or connection never changes your score. Real replies vary, so do not assert on their wording.

# not run here: needs the local runtime, the ollama package and a downloaded model
from ollama import chat

from evidencedesk.llm import LLMRequest, LLMResponse, Usage


class LocalAdapter:
    def __init__(self, model: str) -> None:
        self.model = model  # a model name from the runtime's library page

    def complete(self, request: LLMRequest) -> LLMResponse:
        messages = [{"role": m.role, "content": m.content} for m in request.messages]
        r = chat(model=self.model, messages=messages)
        return LLMResponse(r.message.content, self.model, Usage())  # usage and stop reason: map them only after reading the docs

# not run here: needs the provider SDK, an API key and internet access; without a key it just stops
import os
from pathlib import Path

from openai import OpenAI

from evidencedesk.llm import LLMRequest, LLMResponse, Usage
from rag import Assistant  # your challenge 4 module


class HostedAdapter:
    def __init__(self, model: str) -> None:
        self.client, self.model = OpenAI(), model  # reads OPENAI_API_KEY

    def complete(self, request: LLMRequest) -> LLMResponse:
        system = "\n".join(m.content for m in request.messages if m.role == "system")
        user = "\n".join(m.content for m in request.messages if m.role == "user")
        r = self.client.responses.create(model=self.model, instructions=system, input=user)
        return LLMResponse(r.output_text, self.model, Usage(r.usage.input_tokens, r.usage.output_tokens), "end" if r.status == "completed" else "other")


if "OPENAI_API_KEY" not in os.environ:
    raise SystemExit("no key: this extension is optional and is skipped")
assistant = Assistant(Path("corpus"))
for question in ("How long do I have to return a new bike?", "Can I pay in instalments?"):
    print(question, assistant.ask(HostedAdapter("<model-id>"), question))  # real replies vary: do not assert on them

Provider note (as of 2026-10-12): the local runtime for open-weight models is a separate application; its API listens at http://localhost:11434, the Python call is chat(model=..., messages=[...]) with the text at response.message.content, and the docs showed gemma4 as an example model name. The hosted Responses call is client.responses.create(model=..., instructions=..., input=...) with text in output_text and usage in usage.input_tokens and usage.output_tokens; the docs showed gpt-6-astra as an example ID. Treat both names as examples and copy a current one from the provider's models page. Provider docs advise retrying 429 and 5xx responses with backoff and not retrying 400, 401 or 403, which is the policy of your error classes. The stop-reason values of the local runtime were not checked, so the sketch does not map them. VERIFY — checked 2026-10-12. Check the current documentation.

Submit your work

Before you submit, fill the template files, then run the grader without --no-tests so your own tests run too. Your README says how to run everything offline. Your example output comes from a real run (the reference has a script that regenerates its own). Your design note names three or four trade-offs: what you chose, what you gave up and why, and the limits of your fixtures. A sentence such as "all tests pass" is not a trade-off.

Where this shows up at work

Kwame in Accra joins a team that hires with a take-home task. The reviewer opens the repository and asks the same four questions as this grader: any secret in it, any place where model text becomes a command, any action without a check, any answer that cites something that is not there. Teams that review AI features run these checks as a gate in continuous integration, with a baseline score that may not fall. A candidate who can say "my score was 94, my own fixtures found two misses, and here is the design note" is more convincing than one with a perfect score and no evidence.

Common mistakes

• Coding to the fixtures. A solution can pass by special-casing the questions in the files. Add your own cases and test those; the fixtures are examples.
• Chasing points past a critical failure. Ninety-four points with an eval in the calculator is a fail. Fix the critical item first.
• Trusting the model's citation list. A valid-looking id is not support. Build citations from what you retrieved and check the claim against the cited text.
• Treating the scan as a guarantee. The credential scan finds shapes, not secrets in git history or in a password written in plain words. Keep the manual checklist.
• Placeholder submission files. A README that says TODO fails the submission check. Write them last, from a real run.

Try it yourself

1. Easy. Revisit Reliable Output Contracts. Copy the starter, write only stream.py, and grade it. Predict the score and the result before you run the grader.
2. Medium. Revisit Local Indexes and Retrieval Quality. In a copy of the reference, change the default min_score of search to 0.0. Predict which check fails and what the score and the result are.
3. Hard. Revisit Rubrics, Regression, and Release Gates. Write regress.py: it grades a folder and exits with 1 when the score is below a stored baseline or any critical check fails, even if the submission still passes. Run it on the reference and on the copy from the Medium task.

Answers

Easy. One challenge is worth 15 points, so the score is 15 and the result is FAIL: the pass mark is 70 and the submission files are still placeholders. Save the file below as submission/stream.py.

# file: submission/stream.py
"""Challenge 2: assemble recorded chunks; turn a model reply into a controlled result."""
from evidencedesk.llm import LLMResponse
from evidencedesk.results import parse_answer


def collect(events):
    """events: list of StreamEvent. Returns (status, text); status is complete, empty, interrupted or error."""
    parts = []
    for event in events:
        if event.kind == "chunk":
            parts.append(event.text)
        elif event.kind == "error":
            return "error", "".join(parts)
        elif event.kind == "done":
            text = "".join(parts)
            return ("complete" if text else "empty"), text
    text = "".join(parts)
    return ("interrupted" if text else "empty"), text  # no done event: the stream was cut


def read_reply(response: LLMResponse):
    """Returns (ok, code). Never raises on model output. Codes: ok, truncated, unusable_stop, invalid_json, invalid_schema."""
    outcome = parse_answer(response)
    return outcome.ok, outcome.code

python -m lab.grade submission --verbose | sed -n '/^2\./,/reply_malformed/p;$p'

2. Structured streaming           15/15
     [ok] stream_assemble            4/4
     [ok] stream_interrupted         3/3
     [ok] stream_error               2/2
     [ok] stream_empty               1/1
     [ok] reply_valid                2/2
     [ok] reply_malformed            3/3
SCORE 15/100 | critical failures 0 | RESULT: FAIL

Medium. With no floor, the search returns its best three hits for any query, so a question no document answers is no longer an empty list. Only no_result fails, three points are lost, and the result is still PASS. That is the danger of a pass mark: a real regression hides under it.

cp -r lab/reference copy
sed -i 's/min_score: float = 0.3/min_score: float = 0.0/' copy/search.py
python -m lab.grade copy --verbose --no-tests | grep -E -- '\[--\]|SCORE'

     [--] no_result                  0/3  q7: returned kc-warranty-v2#s2 at 0.26
SCORE 97/100 | critical failures 0 | RESULT: PASS

Hard. The gate below blocks the 97-point copy against a baseline of 100, although 97 passes the lab.

# file: regress.py
"""Block a submission that scores below its stored baseline, even when it still passes (a small release gate)."""
import sys
from pathlib import Path

from lab.grade import grade

folder, baseline = Path(sys.argv[1]), int(sys.argv[2])
report = grade(folder, run_tests=False)
verdict = "BLOCK" if report.critical_failed or report.total() < baseline else "GO"
print(f"{verdict}: {report.total()}/100 (baseline {baseline}), critical failures: {report.critical_failed}")
sys.exit(verdict == "BLOCK")

python regress.py lab/reference 100; echo "exit $?"
python regress.py copy 100; echo "exit $?"

GO: 100/100 (baseline 100), critical failures: []
exit 0
BLOCK: 97/100 (baseline 100), critical failures: []
exit 1

The whole lab is also under test, so a change to the grader cannot silently make it easier (the sed removes the wall-clock time, so the output is the same on every run). The tests cover the reference, the starter, the four flawed copies and seven weakened solutions that each must lose points in the matching check.

python -m pytest tests/test_lab.py | sed 's/ in [0-9.]*s//'

....................                                                     [100%]
20 passed

Pass checklist:

• Easy: you predicted 15 and FAIL, and can say why 15 points are not enough.
• Medium: only no_result fails (3 points), the score is 97 and the result is PASS, and you can explain why that is not safe.
• Hard: GO with exit 0 for the reference and BLOCK with exit 1 for the copy; the script reads the baseline from its arguments and treats critical failures as a block.
• Lab: 70 or more points, 0 critical failures, all required files filled, and your own tests pass offline.

Summary

• The lab turns the ten Parts into six fixture-driven challenges worth 100 points; the pass mark is 70, with all critical checks passed and a complete submission.
• Four critical failures (a committed credential, model text run as code, a bypassed tool authorization, a fake citation shown as verified) outrank the score, and each has an automated check.
• A grader earns trust by catching deliberately failing submissions; the reference solution scored 100 of 100 here.
• The grader checks fixtures only. It cannot show quality on new questions, real models, real users or load, so write your own cases and an honest design note.
• Real-provider extensions are optional and never graded.

Next: Capstone, Cheat Sheet and What Next turns this lab into a finished portfolio project, collects the commands and the architecture on one page, and shows where to go after this tutorial.