Containerisation and Streaming Deployment

Mateo in Bogotá demos the EvidenceDesk assistant on his laptop and it is flawless. In a shared environment the first user closes the tab mid-answer and the model keeps generating for a minute; the image holds a provider key somebody passed as a build argument; the load balancer cuts every stream at a fixed time with no final event; and "healthy" means only that the process exists. (An illustrative case.) No unit test of the answer shows any of this: it lives in the gap between "correct locally" and "safe to run for others".

This page closes that gap for a streaming service: acceptance criteria first, a container, secrets kept out of the image, answers streamed as server-sent events with ids and error events, and tests that a hang-up stops the model work and that overload is refused, not queued without limit.

Honest scope: there is no Docker daemon where this page was written. The Dockerfile and Compose file are shown with # not run here and no output. I checked them statically, with a checker that reads them as text, and tested the behaviours a container relies on (settings, readiness, streaming, hang-ups, shutdown) in Python, once through a real uvicorn on 127.0.0.1. A static check proves what a file says. It cannot prove that an image builds or that Docker stops it as you expect.

What you will learn

• How to write deployment acceptance criteria that point at a test, a static rule or a manual command.
• How to package the service (multi-stage, non-root, health check, exact pins) and keep credentials out of images and source control.
• How to stream with EventSourceResponse: ids, keep-alives, error events, size limits and a secret screen.
• How to test hang-ups, stalls and shutdown, and how to bound in-flight and waiting work.

What is new, and what is reused

No existing file changes. New: the subpackage evidencedesk/serve/, the folder kit/deploy/ (Dockerfile, Compose file, ignore file, pinned runtime requirements, env.example), data/serve/acceptance.json and eight test_deploy_*.py files. Reused: create_app, /health, /ready and the offline pipeline from Packaging, Configuration, and Reproducibility; chunk_text and "only done is complete" from Streaming and Structured Results; UsageLimiter from Budgets, Caching, and Usage Limits; SecretGuard and safe_answer from Safe Failure and Security Regression Tests; EventLog from Structured Logging and Tracing.

Define deployment acceptance criteria

"Done" is a list of claims you can check, in five groups: functionality, security, observability, latency (how long anything may wait) and recovery (a client leaves, the model stalls, the process stops). Each criterion in data/serve/acceptance.json names its check: a test (file and function), a static rule of the container checker, or a manual command that needs a container runtime. acceptance.problems rejects a list with a missing group, a duplicate id, a test that does not exist or an unknown rule, so a criterion cannot point at nothing.

from collections import Counter

from evidencedesk.serve import acceptance

criteria = acceptance.load()
print(dict(Counter(c.category for c in criteria)))
print(acceptance.summary(criteria), "| problems with the list itself:", acceptance.problems(criteria))
r1 = next(c for c in criteria if c.id == "R1")
print(r1.id, r1.kind, r1.ref)
for c in criteria:
    if c.kind == "manual":
        print(c.id, "needs a container runtime:", c.ref.split(" && ")[0])

{'functionality': 5, 'security': 6, 'observability': 2, 'latency': 2, 'recovery': 5}
{'manual': 2, 'static': 5, 'test': 13} | problems with the list itself: []
R1 test test_deploy_disconnect.py::test_hanging_up_stops_the_model_and_the_bill
M1 needs a container runtime: docker build -f deploy/Dockerfile -t evidencedesk-serve:local .
M2 needs a container runtime: docker compose -f deploy/compose.yaml stop

The two manual criteria are exactly the ones that need Docker. This page does not claim them; they stay open. The latency criteria use settings (a 15-second stream limit, a 20-second shutdown window), not measurements: limits you choose, then test.

Package the application with Docker

The image carries the code and the fixture assets read at start (corpus, fixtures, data), nothing else. Dependencies build in one stage and only the finished virtual environment crosses into the runtime stage, which creates a user with a fixed numeric id, switches to it, defines a health check that needs no extra tool, and starts uvicorn in exec form with an explicit shutdown window.

# not run here: needs a Docker daemon. Build context is the kit/ folder:  docker build -f deploy/Dockerfile -t evidencedesk-serve:local .
# Base image: a tag is mutable. For a release build pin the digest of the tag you tested (FROM python:3.12-slim@sha256:<digest>).
FROM python:3.12-slim AS build
ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /build
COPY deploy/requirements-serve.txt .
RUN python -m venv /opt/venv && /opt/venv/bin/pip install -r requirements-serve.txt

FROM python:3.12-slim AS runtime
RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-log-init --no-create-home --shell /usr/sbin/nologin app
COPY --from=build /opt/venv /opt/venv
WORKDIR /app
COPY evidencedesk ./evidencedesk
COPY corpus ./corpus
COPY fixtures ./fixtures
COPY data ./data
ENV PATH=/opt/venv/bin:$PATH PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 EVIDENCEDESK_OFFLINE=1 EVIDENCEDESK_PROVIDER=fake
USER 10001:10001
EXPOSE 8000
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 CMD ["python", "-m", "evidencedesk.serve.probe"]
CMD ["uvicorn", "evidencedesk.serve.asgi:app", "--host", "0.0.0.0", "--port", "8000", "--timeout-graceful-shutdown", "20"]

Tool note (as of 2026-10-12): Docker's best-practice pages describe multi-stage builds (several FROM lines, copy only what you need), USER for a non-root user (with an explicit UID and GID), a .dockerignore with gitignore-like patterns, and tags as mutable (a digest pin gives the same image every build; a tag pin is weaker). HEALTHCHECK has the forms HEALTHCHECK [OPTIONS] CMD command and HEALTHCHECK NONE, defaults interval 30s, timeout 30s, start period 0s, retries 3, and the exit code decides health. Check the current documentation.

Layer and size reasoning, no numbers. I built no image, so I report no size. Structurally: only /opt/venv crosses stages, so build state stays out; requirements-serve.txt (runtime packages only) is copied before the source, so a code change does not reinstall dependencies; and Dockerfile.dockerignore keeps tests, docs, history, caches, .env* and secrets out of the context. It must also keep what the service reads: a rule that drops data gives an image that builds and then reports "not ready" forever, so the checker tests both directions. Measure the real size before you quote one.

Separate build-time and runtime configuration

Build-time facts (base image, pinned requirements) belong in the image. What differs between runs (port, limits, callers, any provider key) belongs to the environment at start. ServeSettings reads only EVIDENCEDESK_* names; a secret is a file path (NAME_FILE) or a value, never both. Two habits do most of the work. A name nobody reads is an error, since MAX_INFLIGHT instead of MAX_IN_FLIGHT would otherwise run silently on the default. A stream limit not shorter than the shutdown window is rejected, since shutdown would cut streams the service could have finished. uvicorn_argv builds the start command from the same settings, and a test parses it with the installed uvicorn, so a misspelt flag fails in tests.

import tempfile
from pathlib import Path

from evidencedesk.serve.settings import load_serve_settings, uvicorn_argv

with tempfile.TemporaryDirectory() as folder:
    secret = Path(folder) / "api_tokens"  # in a container this is /run/secrets/api_tokens
    secret.write_text("nadia:demo-token-0001\n# comment\nomar:demo-token-0002\n", encoding="utf-8")
    env = {"EVIDENCEDESK_API_TOKENS_FILE": str(secret), "EVIDENCEDESK_MAX_IN_FLIGHT": "4", "EVIDENCEDESK_GRACEFUL_S": "25"}
    s = load_serve_settings(env)
print(s.tokens)
print(s.app.offline, s.app.provider, s.app.api_key, s.max_in_flight)
print(" ".join(uvicorn_argv(s)))
bad = [{"EVIDENCEDESK_MAX_INFLIGHT": "4"}, {"EVIDENCEDESK_STREAM_TIMEOUT_S": "30"},
       {"EVIDENCEDESK_API_TOKENS": "a:demo-token-0001", "EVIDENCEDESK_API_TOKENS_FILE": "/x"}]
for env in bad:
    try:
        load_serve_settings(env)
    except ValueError as exc:
        print(list(env)[0][12:], "->", (exc.errors()[0]["msg"] if hasattr(exc, "errors") else str(exc))[:100])

(('nadia', SecretStr('**********')), ('omar', SecretStr('**********')))
True fake None 4
uvicorn evidencedesk.serve.asgi:app --host 0.0.0.0 --port 8000 --timeout-graceful-shutdown 25
_MAX_INFLIGHT -> unknown settings: EVIDENCEDESK_MAX_INFLIGHT
_STREAM_TIMEOUT_S -> Value error, stream_timeout_s must be shorter than graceful_s: shutdown would cut streams that could
_API_TOKENS -> set EVIDENCEDESK_API_TOKENS or EVIDENCEDESK_API_TOKENS_FILE, not both

Tokens print as SecretStr. The image holds no key at all: offline and the fake provider are the defaults, and a real provider needs EVIDENCEDESK_OFFLINE=0 on purpose. Rule D05 of the checker fails any ENV or ARG whose name looks like a key, token, secret, password or credential and which has a value, unless the name ends in _FILE.

Security note (as of 2026-10-12): Docker's documentation says build arguments and environment variables are inappropriate for passing secrets to a build, "because they persist in the final image". An ARG is not in the final container but shows in docker history; an ENV value persists in the image configuration and the running container. Build-time secrets use docker build --secret id=... with RUN --mount=type=secret,id=..., mounted at /run/secrets/<id>. Check the current documentation.

Use Docker Compose secrets where appropriate

For local and single-host use, Compose hands a file to the container. The top-level secrets: entry names the source, the service must list the secret to receive it, and it appears at /run/secrets/api_tokens, the path EVIDENCEDESK_API_TOKENS_FILE points to. The file also sets our hardening: port on 127.0.0.1 only, read-only root, no capabilities, a restart policy and a stop window longer than the 20-second shutdown window. You create deploy/secrets/api_tokens.txt (one name:token per caller) outside version control.

# not run here: needs Docker Compose.  docker compose -f deploy/compose.yaml up --build
services:
  app:
    build:
      context: ..
      dockerfile: deploy/Dockerfile
    image: evidencedesk-serve:local
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      EVIDENCEDESK_OFFLINE: "1"
      EVIDENCEDESK_PROVIDER: fake
      EVIDENCEDESK_API_TOKENS_FILE: /run/secrets/api_tokens
    secrets:
      - api_tokens
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 30s
    restart: unless-stopped
secrets:
  api_tokens:
    file: ./secrets/api_tokens.txt

Tool note (as of 2026-10-12): Compose secrets are "mounted as a file in /run/secrets/<secret_name>"; the source is file or environment, and environment is not supported with docker stack deploy; the how-to says secrets are supported on Linux containers only, each delivered as one bind-mounted file. A _FILE variable such as MYSQL_PASSWORD_FILE is an image convention, not a Docker feature; our service implements it itself. stop_grace_period, read_only, cap_drop and BuildKit reading <Dockerfile name>.dockerignore are not in my research notes: VERIFY them. Check the current documentation.

The limits are real: a mounted file keeps the value out of image layers, but anyone who can read the host file or exec into the container can read it, and it does not rotate. Tokens also identify the caller, so the service no longer trusts a forgeable x-user-id header as the Beginner stand-in did.

Check the container files statically

container.py parses the Dockerfile, loads the Compose file with PyYAML, applies a small subset of .dockerignore matching and returns findings with a rule id and severity. Failures: root, no health check, a secret in ENV/ARG, secrets not as files, secrets in the context, and inconsistent shutdown windows. Warnings are our policy or hygiene: a mutable tag, a writable root, capabilities, cache order.

python -m evidencedesk.serve.container

CONTAINER CHECK (static)
[warn] D01 line 3: tag '3.12-slim' is mutable; pin a digest for a release build
[warn] D01 line 9: tag '3.12-slim' is mutable; pin a digest for a release build
RESULT: PASS (0 fail, 2 warn)

"Pass with two warnings" is honest: a tag is not a digest, and I will not invent one. Now break the files on purpose; each mistake must fail its own rule. (The ignore matcher is a small subset of Docker's rules: a safety net, not a copy.)

import yaml

from evidencedesk.serve.container import check_compose, check_dockerfile, check_ignore

text = Path("kit/deploy/Dockerfile").read_text(encoding="utf-8")
cases = {
    "runs as root": text.replace("USER 10001:10001\n", ""),
    "key in ENV": text.replace("EXPOSE 8000", "ENV EVIDENCEDESK_API_KEY=placeholder\nEXPOSE 8000"),
    "shutdown 5 s, not 20": text.replace('"--timeout-graceful-shutdown", "20"', '"--timeout-graceful-shutdown", "5"'),
}
for name, changed in cases.items():
    print(f"{name:<22}", [f"{f.rule} {f.severity}" for f in check_dockerfile(changed, 20) if f.rule != "D01"])
doc = yaml.safe_load(Path("kit/deploy/compose.yaml").read_text(encoding="utf-8"))
doc["services"]["app"].update(ports=["8000:8000"], stop_grace_period="10s")
print(f"{'compose: open port':<22}", [f"{f.rule} {f.severity}" for f in check_compose(yaml.safe_dump(doc), 20)])
ignore = Path("kit/deploy/Dockerfile.dockerignore").read_text(encoding="utf-8")
print(f"{'ignore without .env*':<22}", sorted({f.rule for f in check_ignore(ignore.replace(".env*", ""))}))

runs as root           ['D03 fail']
key in ENV             ['D05 fail']
shutdown 5 s, not 20   ['D08 fail']
compose: open port     ['C01 fail', 'C06 fail']
ignore without .env*   ['I01']

Implement health and readiness endpoints

/health says the process runs and never depends on data. /ready says the service can do its job: here, that the corpus loaded in the lifespan. create_stream_app wraps the Beginner create_app, so both are reused unchanged. The container health check is a stdlib probe (the slim image has no curl) that exits 0 only when /ready says ready: true; a 503 is unhealthy, not an exception.

from fastapi.testclient import TestClient

from evidencedesk.serve import probe
from evidencedesk.serve.service import create_stream_app
from evidencedesk.serve.testing import rig_settings

app = create_stream_app(rig_settings())
cold = TestClient(app)  # no `with`: the lifespan has not run, nothing is loaded yet
print("before start:", cold.get("/health").json(), cold.get("/ready").status_code, cold.get("/ready").json()["reason"])
with TestClient(app) as http:
    body = http.get("/ready")
    print("after start: ", body.status_code, body.json())
    print("probe says ready:", probe.ready("http://127.0.0.1:8000/ready", lambda url: (body.status_code, body.text)))

before start: {'status': 'alive'} 503 not_loaded
after start:  200 {'ready': True, 'reason': '', 'chunks': 36, 'offline': True}
probe says ready: True

What the check does not know is whether the provider is reachable. Add it as a separate signal later: restarting your container because someone else's service is down causes outages of its own.

Run the application without provider credentials

The default image is the mock mode: offline, fake provider, no key, tokens from a file. The smoke test is a function over a transport, so the same seven steps run in process (below) or against a container over HTTP. Its secret check reads the reassembled answer, since a piece can split a key, as shown shortly.

python -m evidencedesk.serve.smoke

SMOKE
[ok  ] liveness               GET /health -> 200
[ok  ] readiness              GET /ready -> 200
[ok  ] offline mode           offline=True
[ok  ] no token is refused    POST /v1/stream without a token -> 401
[ok  ] stream completes       -> 200, stream complete
[ok  ] answer is cited        a source id is in the text
[ok  ] no secret in output    the made-up key is not in the reassembled answer
RESULT: PASS

Deploy a streaming endpoint

POST /v1/stream (SSE works with any method) returns four event kinds: start, token, error, done. Each has the id request_id:seq and repeats seq in its data, so a client detects a gap, a duplicate or a spliced stream. Only a stream that begins with start, has contiguous ids and ends with done is complete: the rule from Streaming and Structured Results, now with ids. An error carries a stable code, a generic message and retryable, never exception text.

Tool note (as of 2026-10-12): FastAPI documents SSE support added in 0.135.0: response_class=EventSourceResponse from fastapi.sse, items JSON-encoded into data:, a ServerSentEvent class for event, id, retry, and built in a keep-alive comment every 15 seconds when idle, Cache-Control: no-cache and X-Accel-Buffering: no. I used the installed 0.143.0 and saw those headers below. Starlette's TestClient page says httpx2, FastAPI's still says httpx; the kit pins httpx2. Check the current documentation.

Everything refusable is refused before the stream starts, because inside a 200 only an event can be sent. Order: authenticate (so a stranger learns nothing about body validation), resume header (no replay buffer, so Last-Event-ID gets 409), question size (413), readiness (503), the daily quota from UsageLimiter (429, Retry-After to the next UTC midnight), then the admission gate (503). A dependency with yield holds the slot, so it is released when the response ends however it ended.

from evidencedesk.serve.events import judge, parse_stream
from evidencedesk.serve.testing import AUTH, QUESTION

with TestClient(app) as http:
    for headers in ({}, {"authorization": "Bearer wrong-token-0001"}):
        r = http.post("/v1/stream", json=QUESTION, headers=headers)
        print(r.status_code, r.json()["code"], r.headers["www-authenticate"])
    r = http.post("/v1/stream", json=QUESTION, headers=AUTH)
print(r.status_code, r.headers["content-type"], {k: v for k, v in r.headers.items() if k in ("cache-control", "x-accel-buffering")})
print(r.text[:118])
events = parse_stream(r.text)
print(len(events), "events:", events[0].kind, "then", {e.kind for e in events[1:-1]}, "then", events[-1].kind, events[-1].data)
print(judge(events))

401 unauthenticated Bearer
401 unauthenticated Bearer
200 text/event-stream; charset=utf-8 {'cache-control': 'no-cache', 'x-accel-buffering': 'no'}
event: start
data: {"seq": 0, "schema_version": 1}
id: s-1:0

event: token
data: {"seq": 1, "text": "Ref"}
id: s-1:1


11 events: start then {'token'} then done {'seq': 10, 'status': 'answered', 'chunks': 9, 'chars': 127}
Outcome(status='complete', text='Refunds go back to the original payment method within 5 business days after the warehouse receives the item. [kc-returns-v3:c2]', code='')

# excerpt of kit/evidencedesk/serve/service.py (the body of the try in the stream route)
            call = await serve.upstream.start(ticket.question, ticket.user, rid)
            async with aclosing(screened(tallied(call.pieces, tally), guard)) as it:
                while (item := await _next(it, deadline - anyio.current_time())) is not END:
                    if item is None:
                        raise UpstreamError("stream_timeout", True)
                    if serve.poll_disconnect and await request.is_disconnected():
                        return  # the client left: leaving the block closes the call, so the work and the bill stop
                    if chars + len(item) > settings.max_stream_chars or seq > settings.max_events:
                        raise UpstreamError("output_limit", False)
                    yield frame(rid, seq, "token", text=item)
                    seq, pieces, chars = seq + 1, pieces + 1, chars + len(item)
            yield frame(rid, seq, "done", status=call.status, chunks=pieces, chars=chars)
            status = "done"
        except UpstreamError as exc:
            status, code = "error", exc.code
            yield frame(rid, seq, "error", code=exc.code, message="The answer could not be completed.", retryable=exc.retryable)

The excerpt is the body of the try in the route (the code before it sets counters and a deadline). A normal end yields done. A model failure, timeout or cap yields one error inside the open 200. A hang-up cancels the generator at an await; aclosing closes the model call and a finally block (not shown) records the outcome and charges the quota for what was produced, not for the cap. The timeout wraps one __anext__ at a time and never spans a yield, because a cancel scope across a yield in an async generator misbehaves.

The offline model is not incremental: the transport is real, but pieces are cut from a finished, checked answer. A real provider stream differs. You cannot check the whole answer before the first piece, so a secret can arrive split across pieces, and a per-piece scan sees nothing. screened holds back a tail (longest known secret plus a margin) and scans the joined buffer before releasing older text.

# excerpt of kit/evidencedesk/serve/screen.py
async def screened(pieces: AsyncIterator[str], guard: SecretGuard, hold: int | None = None) -> AsyncIterator[str]:
    """Yield text that the guard has seen in full. The last `hold` characters are kept back until more text (or the end) arrives.
    The price is delay: the first `hold` characters of every answer wait. A match raises UpstreamError and nothing of it is sent."""
    hold = hold_for(guard) if hold is None else hold
    buffer = ""
    async with aclosing(pieces) as source:  # closing this generator must close the model call too
        async for piece in source:
            buffer += piece
            if guard.find(buffer):
                raise UpstreamError("secret_screened", False)
            if len(buffer) > hold:
                out, buffer = buffer[:-hold], buffer[-hold:]
                yield out
    if buffer:
        yield buffer

from evidencedesk.security import SIMULATED_SECRET
from evidencedesk.serve.upstream import ScriptedUpstream


def run(upstream):
    app = create_stream_app(rig_settings(), upstream)
    with TestClient(app) as http:
        r = http.post("/v1/stream", json=QUESTION, headers=AUTH)
    return r, judge(parse_stream(r.text)), app.state.serve.records[-1]


cases = [("fails after 6 pieces", ScriptedUpstream(fail_after=6)), ("key split across pieces", ScriptedUpstream(text="See [kc-x] " + SIMULATED_SECRET)),
         ("answer without an end", ScriptedUpstream(text="x" * 5000))]
for name, upstream in cases:
    r, out, rec = run(upstream)
    print(f"{name:<24} {out.status} {out.code:<16} sent={rec.chars:<4} produced={rec.produced:<4} key sent: {SIMULATED_SECRET in out.text + r.text}")

fails after 6 pieces     failed upstream_failed  sent=27   produced=72   key sent: False
key split across pieces  failed secret_screened  sent=0    produced=32   key sent: False
answer without an end    failed output_limit     sent=1995 produced=2052 key sent: False

The price shows in row one: 72 characters produced, 27 sent, because the screen held 45 back when the model failed. That is a deliberate trade, a delay at the start of every answer and a lost tail on failure, for never sending a split key. It guards only strings it knows or can pattern-match; it is not a content filter. Row three is the cap: an error event at about 2,000 characters, not 5,000.

Test client disconnects and timeouts

An abandoned request must not keep spending. Starlette documents request.is_disconnected(); FastAPI's pages say yield-style streaming handles cancellation, and that a task is cancelled only at an await. I assumed neither. testing.call_asgi delivers http.disconnect after N body chunks, with spec_version 2.3 as uvicorn reports; the fake model pauses a millisecond between pieces so cancellation has awaits to land on.

import anyio

from evidencedesk.serve.testing import call_asgi

LONG = "word " * 200  # 1000 characters, 84 pieces


async def tick(_i):
    await anyio.sleep(0.001)


def exchange(hang_up_after=None, **settings):
    up = ScriptedUpstream(text=LONG, pause=tick)
    app = create_stream_app(rig_settings(**settings), up)
    with TestClient(app) as http:
        seen = http.portal.call(lambda: call_asgi(app, "/v1/stream", QUESTION, AUTH, hang_up_after=hang_up_after))
    return seen, up, app.state.serve


for label, k in (("read to the end", None), ("hung up after 8 events", 8)):
    seen, up, serve = exchange(k, max_stream_chars=2000)
    print(f"{label:<24} status={serve.records[-1].status:<9} events seen={len(seen.chunks):<3} produced under a quarter: {up.meter.chars < 250} "
          f"slots held: {serve.gate.in_flight}")


async def stall(i):
    if i == 1:
        await anyio.sleep(3600)  # the model goes silent after the first piece


up = ScriptedUpstream(text=LONG, pause=stall)
app = create_stream_app(rig_settings(stream_timeout_s=0.2), up)
with TestClient(app) as http:
    r = http.post("/v1/stream", json=QUESTION, headers=AUTH)
out = judge(parse_stream(r.text))
print("silent model:", out.status, out.code, "| pieces made:", up.meter.pieces, "| slots held:", app.state.serve.gate.in_flight)

read to the end          status=done      events seen=84  produced under a quarter: False slots held: 0
hung up after 8 events   status=cancelled events seen=8   produced under a quarter: True slots held: 0
silent model: failed stream_timeout | pieces made: 1 | slots held: 0

The hang-up stopped the model at about a tenth of the answer. The tests pin the bound: at most 14 pieces produced after 8 events seen (7 tokens, 4 pieces held by the screen, about two of look-ahead from FastAPI's one-item buffer), and a quota charge equal to what was produced. With the is_disconnected poll removed, nothing changed: under uvicorn's behaviour cancellation arrives anyway. I keep the poll as a flag-controlled second line and do not claim it is required. A generator that never awaits cannot be cancelled, so the upstream must be async. The last row is a timeout: stream_timeout after one piece, no slot held. Keep-alives are FastAPI's own: with its private interval patched from 15 s to 20 ms, an idle stream got : ping comments and still judged complete (that test may need an update after an upgrade).

What TestClient cannot show. Starlette says its client does not emulate concurrent request and response streaming and to use a real server. So one test starts uvicorn on 127.0.0.1 in a thread, reads a few lines from a socket and closes it; another starts a long stream and sets the exit flag with a 1-second graceful window.

python -m pytest kit/tests/test_deploy_realserver.py | tail -n 1 | sed 's/ in [0-9.]*s//'

3 passed

In the shutdown run, uvicorn 0.54.0 logged that it was cancelling one running task because the graceful timeout was exceeded, and the stream record said cancelled with no slot held. That log line is what I observed, not a documented message.

Tool note (as of 2026-10-12): uvicorn's --timeout-graceful-shutdown is the "maximum number of seconds to wait for graceful shutdown. After this timeout, the server will start terminating requests"; uvicorn finishes in-flight responses and background tasks within the configured timeouts. The pages read do not say what SIGTERM does, so I make no claim about it; how docker stop signals and then kills was not confirmed either: VERIFY it and test with your own stream (criterion M2). The uvicorn.workers module is documented as deprecated, so avoid gunicorn -k uvicorn.workers.UvicornWorker for new work. Hosting platforms differ on injecting a PORT variable; read their pages. Check the current documentation.

Define concurrency and queue limits

Each open stream holds a connection, memory and a model call, so unbounded concurrency is a bill and an outage in waiting. AdmissionGate(max_in_flight, max_waiting, max_wait_s) bounds both: streams past max_in_flight may wait, but only max_waiting of them and only max_wait_s; the rest get 503 with Retry-After at once. The default is no waiting room, since a waiting client sees no events and a fast refusal is usually kinder. Choose numbers from measured capacity, not from this page. The design is ours; HTTP says only that 503 means temporary overload and may carry Retry-After, and 429 (RFC 6585) means too many requests in a given time.

release = anyio.Event()


async def hold(i):
    if i == 1:
        await release.wait()  # the first stream stays open until we let it finish


app = create_stream_app(rig_settings(max_in_flight=1, max_waiting=0), ScriptedUpstream(pause=hold))


async def scenario():
    got = {}

    async def first():
        got["first"] = await call_asgi(app, "/v1/stream", QUESTION, AUTH)

    async with anyio.create_task_group() as tg:
        tg.start_soon(first)
        while app.state.serve.gate.in_flight < 1:
            await anyio.sleep(0.001)
        got["busy"] = await call_asgi(app, "/v1/stream", QUESTION, AUTH)
        release.set()
    return got


with TestClient(app) as http:
    got = http.portal.call(scenario)
gate = app.state.serve.gate
print("busy:", got["busy"].status, "retry-after", got["busy"].headers["retry-after"], got["busy"].text)
print("first:", judge(parse_stream(got["first"].text)).status, "| rejected:", gate.rejected, "| in flight:", gate.in_flight)

busy: 503 retry-after 1 {"code":"overloaded","message":"The service is busy: try again shortly."}
first: complete | rejected: 1 | in flight: 0

The refused request cost no model call, the first stream finished unharmed and the slot count returned to zero (a test also covers recovery afterwards and the bounded waiting room). The next page builds on this gate.

Exercise: a containerised offline service

With Docker, build, start with a tokens file, smoke-test from the host, and stop with a stream open. Nothing below was run for this page.

# not run here: needs a Docker daemon and Compose (criteria M1 and M2)
mkdir -p kit/deploy/secrets
echo "nadia:local-demo-token-1" > kit/deploy/secrets/api_tokens.txt
docker compose -f kit/deploy/compose.yaml up --build -d
python -m evidencedesk.serve.smoke http://127.0.0.1:8000 kit/deploy/secrets/api_tokens.txt
docker compose -f kit/deploy/compose.yaml ps
docker compose -f kit/deploy/compose.yaml stop

What the files predict, not what a run showed: ps reports healthy once /ready answers, the smoke test prints the same seven steps, and stop returns within the stop window, the open stream ending as an interrupted stream. If your terminal disagrees, believe it and fix the file or the criterion. Without Docker the same checks run in process:

python -m pytest kit/tests/test_deploy_*.py | tail -n 1 | sed 's/ in [0-9.]*s//'

103 passed

These tests prove code paths and file contents. They cannot prove that an image builds, that a load balancer in front of you does not buffer or cut streams, or that your capacity numbers are right.

Where this shows up at work

Omar in Cairo inherits an assistant that "worked in staging". In a week he adds a readiness probe, a mounted key file, stream and in-flight caps and an acceptance list. The first load-balancer incident, streams cut with no final event, takes minutes to diagnose because clients can tell an interrupted stream from a finished one. (An illustrative case.)

Common mistakes

• Passing a key as ARG or ENV. It lives in image history or configuration. Mount a file, name it *_FILE, and let the checker fail the build.
• Running as root on a writable filesystem. A bug becomes a takeover. Use a numeric non-root user, a read-only root and a tmpfs.
• One "healthy" for everything. Liveness that checks the corpus restarts a container that needs time; readiness that checks nothing sends traffic to an empty service.
• Letting a stream end silently. A cut stream looks like a short answer. Send ids and a final done; treat anything else as interrupted.
• Scanning each piece for secrets. A split key passes. Scan a joined window and accept its delay.
• Unlimited queues. They hide overload until memory is gone. Bound in-flight and waiting work; refuse the rest with Retry-After.

Try it yourself

1. Easy. Revisit Streaming and Structured Results. Cut a real stream after five events and ask judge what it is. Predict, then compare with the whole stream.
2. Medium. Revisit Packaging, Configuration, and Reproducibility, whose REQUIRED_DATA the service needs at start. Add data to the ignore file and predict which rule fails and why that is worse than a failed build.
3. Hard. Revisit Structured Logging and Tracing. Run one finished stream, one hung-up stream and one rejected request on a fresh app, count outcomes from the log only, and show the question text is in no line.

Answers

Easy. Five events are a start and four tokens with no done: interrupted; the whole stream is complete.

with TestClient(create_stream_app(rig_settings())) as http:
    wire = http.post("/v1/stream", json=QUESTION, headers=AUTH).text
cut = "\n\n".join(wire.split("\n\n")[:5]) + "\n\n"  # the connection drops after five events
print(len(parse_stream(cut)), "events arrived ->", judge(parse_stream(cut)).status, "| whole stream ->", judge(parse_stream(wire)).status)

5 events arrived -> interrupted | whole stream -> complete

Medium. Rule I02. The build would succeed and the container would start, then report not ready forever (the loader finds no corpus): an orchestrator sees a crash loop, not a bad build.

from evidencedesk.serve.container import check_ignore

print([(f.rule, f.message[:62]) for f in check_ignore(ignore + "\ndata\n")])

[('I02', "the ignore file drops 'data', which the service reads at start")]

Hard. One stream.start per stream, outcomes done and cancelled, and one rejection with its code. The log keeps question_chars, never the text, and callers appear as pseudonyms.

from collections import Counter

app = create_stream_app(rig_settings())
with TestClient(app) as http:
    http.portal.call(lambda: call_asgi(app, "/v1/stream", {"question": "Can I pay in instalments?"}, AUTH))
    http.portal.call(lambda: call_asgi(app, "/v1/stream", QUESTION, AUTH, hang_up_after=3))
    http.post("/v1/stream", json=QUESTION, headers={"authorization": "Bearer wrong-token-0001"})
records = app.state.serve.log.records()
print(sorted(Counter((r["event"], r["outcome"] or r["fields"].get("code", "")) for r in records).items()))
print("question text in any line:", any("instalments" in line for line in app.state.serve.log.lines))

[(('request.rejected', 'unauthenticated'), 1), (('stream.end', 'cancelled'), 1), (('stream.end', 'done'), 1), (('stream.start', ''), 2)]
question text in any line: False

Pass checklist:

• Easy: interrupted for 5 events, complete for the whole stream, and the reason (no done).
• Medium: I02, explained as a start-time failure, not a build failure.
• Hard: two starts, one done, one cancelled, one unauthenticated, and no question text in any line.

Summary

• Write acceptance criteria first, each tied to a test, a static rule or a visible manual step; keep the container-only ones on the list as open.
• Keep the image small, pinned, non-root and secret-free; pass secrets as files; fail on unknown settings.
• Separate liveness from readiness; health-check with tools already in the image.
• Stream events with ids, a start and a done, error events with codes, caps and keep-alives; refuse before the stream starts; screen a joined window for secrets.
• Test hang-ups, stalls and shutdown with raw ASGI and one real server; bound in-flight and waiting work and answer overload with 503 and Retry-After.
• Nothing here ran in a container: the checker and tests prove files and behaviours, and the manual criteria stay open until you run them.

Next: Scaling, Resilience, and Rollback separates stateless workers from durable state, adds backpressure, retries with idempotency keys, rate limits, timeouts across the request chain, provider-outage fallbacks, environment separation, a rollback plan and a fake-client load test.