Scaling, Resilience, and Rollback

Priya in Pune ships the streaming EvidenceDesk service from the last page. A campaign triples traffic. The model provider slows, clients time out and retry, and the retries deepen the slowdown. That night a new prompt version goes to everyone at once; answers come back truncated and nobody knows how many users saw them or how to go back. (An illustrative case.) Each failure is ordinary; having no plan is what hurts. This page is that plan, in the order a request meets it.

Honest scope: there is no Docker daemon or orchestrator where this page was written, and I state no cloud-provider facts. Everything here is a simulation in virtual time with fake providers. Simulations show mechanics, not production numbers. All prices, rates and capacities are invented.

What you will learn

• Why concurrency, not parallelism, serves IO-bound model calls, what workers share, and how Little's law predicts a load test.
• How to bound overload, timeouts and retries so one slow dependency cannot take the service down.
• How to fail over between providers without an untested model, with breakers and bulkheads.
• How to release by canary with metric gates, roll back by flag or plan, and rehearse in a game day.

What is new, and what is reused

New in evidencedesk/serve/: ten modules, three data files in data/serve/ and seven test_scale_*.py files. Reused: AdmissionGate and ServeSettings (Containerisation and Streaming Deployment), UsageLimiter (Budgets, Caching, and Usage Limits), RetryPolicy (Adapters, Configuration, and Error Handling), storm, Breaker, Flags and the alert Engine (Monitoring, Budgets, and Incidents) and the release gate (Robust Metrics and Release Gates).

Concurrency, parallelism and the worker model

A model call spends nearly all its time waiting. Concurrency means many requests in progress at once, most waiting; parallelism means several running on several cores at the same instant. An IO-bound service needs the first; parallelism serves only the small computing part (retrieval, screening, JSON). Here a request holds a core for 15 ms and waits 800 ms for the model, and a worker has in-flight slots (AdmissionGate.max_in_flight). The ceiling is the smaller of slots / (cpu + io) and cores / cpu.

from evidencedesk.serve.loadsim import Profile, Service, capacity, simulate, sweep

p = Profile(rate=1, cpu_s=0.015, io_s=0.8)  # one request: 15 ms on a core, 800 ms waiting for the model
for name, s in {"1 worker, 8 slots": Service(1, 8), "1 worker, 64 slots": Service(1, 64), "4 workers, 64 slots": Service(4, 64)}.items():
    print(f"{name:<20} ceiling {capacity(p, s):6.1f} req/s")

1 worker, 8 slots    ceiling    9.8 req/s
1 worker, 64 slots   ceiling   66.7 req/s
4 workers, 64 slots  ceiling  266.7 req/s

Eight slots cap a worker at 9.8 requests per second while its core is nearly idle. Sixty-four slots lift that and the core becomes the limit; only more processes lift that (parallelism). Raise concurrency first, add workers when the CPU is the limit. Workers share nothing unless you arrange it: each has its own AdmissionGate (four workers at max_in_flight=8 admit 32), cache, usage counters, breaker and flags.

Tool note (as of 2026-10-12): uvicorn's --workers defaults to $WEB_CONCURRENCY if set, else 1, and is not valid with --reload. Its workers are spawned, not pre-forked, and restarted if they die; SIGHUP restarts them one at a time. The uvicorn.workers module is deprecated: prefer uvicorn --workers N or the uvicorn-worker package. Gunicorn's --graceful-timeout defaults to 30 s, after which workers are force-killed. What uvicorn does on SIGTERM is not stated on the pages read: test it. Check the current documentation.

Separate stateless workers from durable state

A worker is stateless when losing it, adding one or sending the next request elsewhere changes no answer; then you can scale and restart it freely. State does not vanish, it moves somewhere durable.

• History and agent checkpoints: a database by user and session; in memory they vanish on restart or on another worker.
• Usage limits: one shared counter; per worker, the limit multiplies by the worker count.
• Response cache: a shared cache, or accept the loss; per worker, the hit rate falls and model calls rise.
• Admission gate and breaker: per process on purpose, since they protect this process; size them per worker.

Cluster sends requests round-robin to four workers: with shared=False each keeps its own state; with shared=True all use one object that stands for a persistent service (its atomicity and failures are not modelled).

from evidencedesk.limits import Quota
from evidencedesk.serve.placement import Cluster
from evidencedesk.usage import ManualClock

clock = ManualClock(0.0)
for shared in (False, True):
    c = Cluster(4, shared, clock)  # four workers; each user may make 10 calls a day
    c.ask("aiko", "refund window")
    seen = c.ask("aiko", "and sale items?")["history_seen"]
    allowed = sum(c.ask("omar", f"q{i}")["status"] == 200 for i in range(40))
    big = Cluster(4, shared, clock, per_user=Quota(1000, 10**6))
    hits = sum(big.ask("mateo", q)["cache"] == "hit" for q in ["refund window", "shipping time"] * 10)
    print(f"shared={shared!s:<5} follow-up saw {seen} earlier question(s) | limit 10 let {allowed}/40 through | cache hits {hits}/20, model calls {big.model_calls}")

shared=False follow-up saw 0 earlier question(s) | limit 10 let 40/40 through | cache hits 16/20, model calls 4
shared=True  follow-up saw 1 earlier question(s) | limit 10 let 10/40 through | cache hits 18/20, model calls 2

Separate counters let a user make 40 calls under a limit of 10; a follow-up lands on a worker with no history; the cache does twice the model calls. Sharing adds a dependency on the critical path, which needs its own timeout and fallback.

Load-test with a fake client

A load test asks "what does the service do at rate R?" without spending tokens, so generator and service are both fake and run in virtual time: simulate(Profile, Service) draws seeded Poisson arrivals, runs a discrete-event queue and returns a LoadReport, with no sleeping and no sockets. Arrivals are open-loop: clients keep coming however slow the service is, as in a campaign (a closed-loop generator slows down with the service and hides overload). One worker, 8 slots, 8 places to wait:

print("rate  shed%  p95_s  done/s  slots_used  L=in system  lambda*W")
for rate, r in sweep([4, 8, 10, 14], Service(1, 8, max_waiting=8), p):
    print(f"{rate:>4}  {100 * r.shed / r.offered:5.1f}  {r.p95_s:5.2f}  {r.throughput:6.2f}  {r.util:10.2f}  {r.in_system:11.2f}  {r.throughput * r.mean_s:8.2f}")

rate  shed%  p95_s  done/s  slots_used  L=in system  lambda*W
   4    0.0   1.19    4.22        0.43         3.46      3.46
   8    0.9   1.44    8.06        0.82         7.51      7.51
  10    6.3   1.72    9.49        0.96        11.09     11.09
  14   30.4   1.91    9.88        1.00        14.35     14.35

This is a saturation curve. Past the 9.8 ceiling, p95 rises, requests are shed and throughput flattens. The last two columns are Little's law, L = λ × W: the average number of requests in the system equals the arrival rate times the average time each spends there. L is integrated from the event stream and λ × W computed from completions, so their agreement checks the simulator. It also plans capacity: 40 requests per second of 0.815 s each keep about 33 in flight.

The simulator has one perfect queue and no network: it shows the shape of saturation. Measure the real service with a real load tool and a fake model, so the test costs nothing.

Handle backpressure and overload

Backpressure is the service saying "slower" instead of silently absorbing work. Without it a queue grows without bound, latency climbs, and every answer arrives after the client gave up. Three designs against the same overload (14 per second, ceiling 9.8):

shapes = {"no waiting room": Service(1, 8), "8 may wait": Service(1, 8, max_waiting=8),
          "unbounded, 3 s deadline": Service(1, 8, max_waiting=10_000, deadline_s=3.0)}
for name, s in shapes.items():  # the same 14 requests per second against a ceiling of 9.8
    r = simulate(Profile(14), s)
    print(f"{name:<24} shed {r.shed:>4}  expired {r.expired:>4}  late {r.late:>4}  on time {r.ok:>4}  p95 {r.p95_s}s")

no waiting room          shed  696  expired    0  late    0  on time 1028  p95 1.172s
8 may wait               shed  524  expired    0  late    0  on time 1200  p95 1.912s
unbounded, 3 s deadline  shed    0  expired  503  late 1158  on time   63  p95 4.03s

With no waiting room, 696 requests are refused at once and the rest are fast. Eight places to wait refuse fewer (524) at the price of a higher p95. The unbounded queue with a 3-second deadline refuses nobody and serves almost nobody: 503 requests waited out their deadline, 1,158 were processed after it (work done for nobody, and billed), and 63 were on time. Bound the queue, drop expired work at the front, and answer overload at once with a retry hint. AdmissionGate from the last page is this design; the simulator lets you pick its numbers by experiment.

Tool note (as of 2026-10-12): HTTP 503 means "currently unable to handle the request due to a temporary overload or scheduled maintenance" and may carry Retry-After; some servers simply refuse the connection. HTTP 429 (RFC 6585) means "too many requests in a given amount of time". Shedding advice is our own design. Check the current documentation.

Configure timeouts across the request chain

Every hop has a timeout, and they must nest: each layer gives up before the one above, or the upper layer abandons work that goes on unseen (an orphaned, billed model call). chain_problems checks this against ServeSettings. A Deadline carries the budget with the request: each hop gets what is left, and a caller header may shorten it, never lengthen it. call_with_deadline never starts a call after the deadline, caps each try at the time left, does not retry what retrying cannot fix, and stops when a wait would pass the deadline.

import random

from evidencedesk.errors import LLMError, RetryPolicy
from evidencedesk.serve.chain import Deadline, call_with_deadline, chain_problems, layers_from
from evidencedesk.serve.settings import ServeSettings

for name, args in {"as configured": (30, 12, 3, 10, 2), "repaired": (30, 20, 2, 5, 1)}.items():  # client, gateway, tries, seconds per try, backoff
    chain = layers_from(ServeSettings(), *args)  # the stream limit is 15 s
    print(name, [(layer.name, layer.timeout_s) for layer in chain])
    for problem in chain_problems(chain):
        print("  -", problem)

policy = RetryPolicy(max_attempts=4, base_delay=0.5, factor=2.0, max_delay=8.0, jitter=1.0)  # jitter=1.0: wait anywhere in 0..backoff
for label, plan, seconds in (("two timeouts, then ok", ["timeout", "timeout"], 12), ("429, Retry-After 4 s, 3 s left", ["rate_limited"], 3), ("invalid request", ["invalid_request"], 12)):
    clock, tries, kinds = ManualClock(0.0), [], iter(plan)

    def model(timeout):  # a fake model call that fails as the plan says
        tries.append(round(timeout, 1))
        if kind := next(kinds, None):
            clock.advance(timeout if kind == "timeout" else 0.1)
            raise LLMError(kind, "simulated", retry_after=4.0 if kind == "rate_limited" else None)
        return "answer"
    try:
        outcome = call_with_deadline(model, Deadline.after(clock, seconds), policy, attempt_s=5.0, sleep=clock.advance, rng=random.Random(1))
    except LLMError as error:
        outcome = f"gave up: {error.kind}"
    print(f"{label:<36} timeouts given {tries} -> {outcome}")

as configured [('client', 30), ('gateway', 12), ('stream', 15.0), ('model client', 32)]
  - stream waits 15.0s but gateway waits only 12s: gateway gives up first
  - model client waits 32s but stream waits only 15.0s: stream gives up first
repaired [('client', 30), ('gateway', 20), ('stream', 15.0), ('model client', 11)]
two timeouts, then ok                timeouts given [5.0, 5.0, 1.4] -> answer
429, Retry-After 4 s, 3 s left       timeouts given [3.0] -> gave up: rate_limited
invalid request                      timeouts given [5.0] -> gave up: invalid_request

The configured chain fails twice: the gateway gives up at 12 s while the stream may run 15 s, and the model client may retry for 32 s inside a 15 s stream. In the runs, the third try gets only the 1.4 s left; a 429 whose Retry-After (4 s) exceeds the 3 s left is returned at once; a caller's mistake is tried once.

Design retry and idempotency behaviour

Retries turn short failures into recoveries and long ones into outages. Three controls keep them useful: backoff with jitter (wait longer each time, by a random amount, so clients do not return in step), a retry budget (retries add at most 10% of first tries) and honouring Retry-After.

Provider note (as of 2026-10-12): an AWS Architecture Blog post on retries names "Full Jitter", "Equal Jitter" and "Decorrelated Jitter", gives Full Jitter as random_between(0, min(cap, base * 2 ** attempt)) and says that without jitter retries cluster. RetryPolicy(jitter=1.0) here is Full Jitter. Retry-After is an HTTP-date or seconds; MDN says support is "still inconsistent". "Honour it if present, else full jitter, cap attempts, retry only what is safe" is our own advice. Check the current documentation.

The storm simulation from Monitoring, Budgets, and Incidents (12 new requests per second, capacity 20, a 30-second outage) shows each control:

from evidencedesk.observe.resilience import BACKOFF, NAIVE, storm

for name, policy, budget in (("retry at once x4", NAIVE, None), ("backoff + full jitter", BACKOFF, None), ("same, plus a 10% retry budget", BACKOFF, 0.1)):
    s = storm(policy, retry_budget=budget)  # 12 new requests/s, capacity 20/s, a 30 s outage
    print(f"{name:<30} calls {s.attempts:>4}  peak {s.peak:>2}/s  served {s.served}  gave up {s.gave_up}  seconds over capacity {s.overloaded_seconds}")

retry at once x4               calls 2188  peak 48/s  served 756  gave up 324  seconds over capacity 33
backoff + full jitter          calls 1814  peak 38/s  served 746  gave up 334  seconds over capacity 32
same, plus a 10% retry budget  calls 1130  peak 24/s  served 722  gave up 358  seconds over capacity 1

Retrying at once multiplies the load by four (peak 48 calls per second against 12 new requests and capacity 20). Jitter lowers the peak. The 10% budget keeps the provider within capacity for all but one second, at the price of giving up on a few more requests. Served stays near 750 in all three runs: the outage limits it, and retries cannot create capacity.

Idempotency. A client times out but the server finished; the retry repeats the work: tokens for a read, a second ticket for "create a ticket". An idempotency key, one per intended action, is stored with a fingerprint of the request and, when done, the response. A repeat returns the stored response; the same key with a different request is an error. IdempotencyStore.begin uses one atomic INSERT OR IGNORE to pick which of two racing requests runs; a failure abandons the key so a retry may run; keys expire.

import anyio

from evidencedesk.serve.dedupe import IdempotencyStore, SingleFlight, handle_once

tickets = []

def create(order):
    def work():
        tickets.append(order)
        return f"ticket {len(tickets)} for {order}"
    return work

store = IdempotencyStore(clock)  # sqlite in memory stands in for a shared database
for label, key, order in (("first call", "k-1", "order 1042"), ("retry after a timeout", "k-1", "order 1042"), ("same key, other order", "k-1", "order 7"), ("new key", "k-2", "order 7")):
    print(f"{label:<22}", handle_once(store, key, order, create(order)), "| tickets:", len(tickets))

async def burst():
    flight, calls = SingleFlight(), []
    async def model():
        calls.append(1)
        await anyio.sleep(0.01)
        return "answer"
    async def one():
        await flight.do("refund window", model)
    async with anyio.create_task_group() as tg:
        for _ in range(10):
            tg.start_soon(one)
    print("10 identical requests ->", len(calls), "model call,", flight.followers, "followers")
anyio.run(burst)

first call             (200, 'ticket 1 for order 1042') | tickets: 1
retry after a timeout  (200, 'ticket 1 for order 1042') | tickets: 1
same key, other order  (422, 'idempotency_key_reused') | tickets: 1
new key                (200, 'ticket 2 for order 7') | tickets: 2
10 identical requests -> 1 model call, 9 followers

The retry returns ticket 1 instead of creating ticket 2; the same key for another order is refused with 422. The last line is the sibling control, single-flight: ten identical in-flight questions share one model call, which matters most when a cached answer expires under load.

Add rate limits and per-user budgets

Two limits protect two things. A rate limit (a token bucket: refill rate and burst) protects capacity over seconds. A budget (UsageLimiter, from Budgets, Caching, and Usage Limits) protects money over a day; BudgetKeeper from Monitoring, Budgets, and Incidents adds degrade-before-block. A rejection says when to return: the bucket computes Retry-After. Injected clocks make the tests exact.

from evidencedesk.limits import UsageLimiter
from evidencedesk.serve.dedupe import PerUserRate

clock = ManualClock(0.0)
rate, daily = PerUserRate(2.0, 3, clock), UsageLimiter(clock, Quota(5, 10_000), Quota(100, 10**6))  # 2 calls/s, burst 3; 5 calls/day
seen = []
for t in (0, 0, 0, 0, 0.5, 1.0, 1.0, 1.0):
    clock.now = t
    ok, wait = rate.take("nadia")
    seen.append("ok" if ok else f"429 (retry in {wait} s)")
print(seen)
print("omar has his own bucket:", rate.take("omar"))
day = []
for _ in range(7):
    day.append(daily.check("omar", 100).code)  # the five-call daily budget
    daily.record("omar", 100)
print(day)

['ok', 'ok', 'ok', '429 (retry in 1 s)', 'ok', 'ok', '429 (retry in 1 s)', '429 (retry in 1 s)']
omar has his own bucket: (True, 0)
['ok', 'ok', 'ok', 'ok', 'ok', 'budget_exceeded', 'budget_exceeded']

Three calls use the burst, the fourth is refused with a 1-second hint, half a second refills one token, and another user has a bucket of their own. The daily budget allows five calls, then answers budget_exceeded. Both counters must be shared across workers.

Plan for provider outages

A circuit breaker opens after consecutive failures so the service fails fast instead of hammering a dead provider, then allows one trial call after a reset time (half-open). A bulkhead gives each tenant, route or provider its own compartment. Failover sends a failed call to another provider. Explicit degradation says plainly what the user gets when nothing works.

Failover is the dangerous one: a second provider is a different system. Three rules in Router.complete: the fallback must have been through the same release gate without a block, so an untested model is never an automatic fallback; it must be allowed to receive the data in the request; and the response must say who served it (served_by, tried). A caller's mistake is raised, not retried elsewhere. The course has one fake model, so each provider is stood in for by a retrieval configuration scored by the real gate: the mechanism is the point, not the scores.

from evidencedesk.evalkit.suite import Config
from evidencedesk.llm import LLMRequest, Message
from evidencedesk.serve.failover import FakeProvider, Router
from evidencedesk.serve.rollout import preflight

stand_ins = {"main": Config("sentence", 0.35, 0.05), "backup-a": Config("paragraph", 0.35, 0.05), "backup-b": Config("sentence", 0.2, 0.05)}
verdicts = {name: preflight(cfg).verdict for name, cfg in stand_ins.items()}  # the release gate of the evaluation pages, per provider
print(verdicts)
approved = frozenset(n for n, v in verdicts.items() if v != "block")
request = LLMRequest((Message("user", "How long do refunds take?"),))

def providers(main_up=True, a_up=True):  # an outcome script: None is success, a string is the error kind
    return [FakeProvider("main", outcomes=lambda n: None if main_up else "server"),
            FakeProvider("backup-a", allows=frozenset({"public", "internal"}), outcomes=lambda n: None if a_up else "timeout"),
            FakeProvider("backup-b")]

for label, up, data in (("main down", {"main_up": False}, "internal"), ("main down, personal data", {"main_up": False}, "personal"),
                        ("main and backup-a down", {"main_up": False, "a_up": False}, "internal")):
    out = Router(providers(**up), approved, ManualClock(0.0)).complete(request, data)
    print(f"{label:<26} served_by={out.served_by or '-':<9} degraded={out.degraded!s:<5} {[f'{n}:{o}' for n, o in out.tried]}" + (f" -> {out.payload['code']}" if out.payload else ""))

{'main': 'review', 'backup-a': 'review', 'backup-b': 'block'}
main down                  served_by=backup-a  degraded=True  ['main:server', 'backup-a:ok']
main down, personal data   served_by=-         degraded=False ['main:server', 'backup-a:skipped:data_class', 'backup-b:skipped:not_approved'] -> temporarily_unavailable
main and backup-a down     served_by=-         degraded=False ['main:server', 'backup-a:timeout', 'backup-b:skipped:not_approved'] -> temporarily_unavailable

backup-b is blocked by the gate and never called, even when everything else is down: unavailable beats unmeasured. With personal data the approved fallback is skipped for its data class, so the user gets a clear temporary failure.

from evidencedesk.serve.admission import Overloaded
from evidencedesk.serve.failover import Bulkheads

router = Router(providers(main_up=False), approved, clock)
trail = []
for _ in range(5):
    trail.append((router.breaker("main").state, router.complete(request).tried[0][1]))
print(trail)
clock.advance(31)
print("31 s later:", router.breaker("main").state)

async def flood(gate, n):
    admitted = 0
    for _ in range(n):
        try:
            await gate.enter()  # the calls never finish: a tenant that floods
            admitted += 1
        except Overloaded:
            pass
    return admitted

async def compare():
    for name, key in (("one shared gate", lambda tenant: "all"), ("a gate per tenant", lambda tenant: tenant)):
        bulk = Bulkheads(4)
        print(f"{name:<18} tenant-a got {await flood(bulk.gate(key('a')), 20)} of 20, tenant-b got {await flood(bulk.gate(key('b')), 2)} of 2")
anyio.run(compare)

[('closed', 'server'), ('closed', 'server'), ('closed', 'server'), ('open', 'skipped:breaker_open'), ('open', 'skipped:breaker_open')]
31 s later: half_open
one shared gate    tenant-a got 4 of 20, tenant-b got 0 of 2
a gate per tenant  tenant-a got 4 of 20, tenant-b got 2 of 2

The breaker fails fast after three failures and is half-open after the reset time. The bulkhead run uses the real AdmissionGate: with one shared gate tenant-a's flood starves tenant-b; with a gate per tenant tenant-b is untouched.

Separate development, staging and production settings

The same image runs everywhere; the environment differs. Keep each environment's settings as data and check them with rules. envs.check loads each set with the real load_serve_settings (a misspelt name fails) and adds: tokens and keys come from files, not values (E01, E02); each environment's secret files sit in its own folder, so one leaked file opens one environment (E03); the offline fake is chosen on purpose (E04); production cannot run the fake, an unnamed model or a demo caller (E05 to E07). drift lists what differs between two environments.

import copy

from evidencedesk.serve import envs

all_envs = envs.load_envs()
print({name: [f.rule for f in envs.check(name, all_envs)] for name in all_envs})
copied = copy.deepcopy(all_envs)
copied["production"] = dict(all_envs["dev"])  # production started from the dev file
for f in envs.check("production", copied):
    print(f.rule, f.text[:84])
print("staging vs production differ in:", list(envs.drift(all_envs["staging"], all_envs["production"])))

{'dev': [], 'staging': [], 'production': []}
E01 callers' tokens must come from a file (API_TOKENS_FILE), not from the environment
E05 production must not run the offline fake
E06 production must name its model: a default can change under you
E07 a test or demo caller name is configured
staging vs production differ in: ['EVIDENCEDESK_API_TOKENS_FILE']

Production built from the dev file fails four rules (a secrets file shared with staging would fail E03). The files hold paths to secrets, never secrets. Debug switches are the same problem as test credentials, which is why unknown setting names are errors.

Release strategies, metric gates and rollback

A release is an experiment on users. Rolling replaces instances in steps; a canary sends a small share to the new version and widens it while the numbers hold; blue-green prepares a second full fleet and flips all traffic at once, so flipping back is the rollback. Gate every step: before traffic, the offline gate (preflight: go, review or block); during the rollout, the new version's requests against the old one's from the same window.

decide(canary, baseline) returns proceed, hold or rollback with reasons. Too few requests is hold, never proceed. The error rule uses the Wilson interval from Robust Metrics and Release Gates: roll back only when even the low end of the new error rate is a margin above the old rate, so luck on a small sample neither stops a good release nor passes a bad one. Validation failures use the same rule; p95 and cost use ratios. Cohorts are sticky: in_cohort hashes the user, so a user stays on the new version as the share grows.

The scenario: version 2 adds a lookup that fails more as more traffic uses it. Each strategy runs with the gate off (the alerts of Monitoring, Budgets, and Incidents find it; a person rolls back ten minutes later) and on:

from evidencedesk.serve.rollout import STRATEGIES, load_scenario, run_rollout

sc = load_scenario()  # v2 adds a lookup that fails more as more traffic uses it (data/serve/gameday.json)
print(f"{'strategy':<11} {'gate':<5} {'outcome':<12} {'v2 requests':>11} {'bad':>5} {'back at':>8}")
for strategy in STRATEGIES:
    for gated in (False, True):
        r = run_rollout(sc, strategy, gated)
        print(f"{strategy:<11} {'on' if gated else 'off':<5} {r.outcome:<12} {r.exposed:>11} {r.failed_exposed:>5} {r.rollback_at_s / 60:>5.0f} min")

strategy    gate  outcome      v2 requests   bad  back at
rolling     off   rolled_back         4714  1441   125 min
rolling     on    rolled_back          293    32    30 min
canary      off   rolled_back         2551   848    65 min
canary      on    rolled_back          451    69    30 min
blue_green  off   rolled_back         2100   789    35 min
blue_green  on    rolled_back          600   217    10 min

Gate off, the bad release reached 4,714 requests with rolling and 2,551 with a canary, and an alert found it about two hours in. Gate on, exposure fell to 293 and 451. Blue-green exposes everyone at once (600 requests in ten minutes) but rolls back fastest, by flipping. None of the three finds a defect that appears only at scale unless the gate keeps looking at every step.

Rollback triggers, flags and the kill switch

from evidencedesk.observe.resilience import Flags, choose_mode

flags = Flags(ManualClock(0.0), values={"answer.enabled": True, "answer.mode": "auto", "prompt.version": "v1", "rollout.share": 0.0})
r = run_rollout(sc, "canary", True, flags)
for s in r.steps:
    print(f"t={s.t_s / 60:>3.0f} min  v2 share {s.share:>4.0%}  v2 errors {s.canary_errors:>2}/{s.canary_n:<3}  {s.action:<8} {'; '.join(s.reasons)}")
print([(h["from"], h["to"]) for h in flags.history], "by", flags.history[-1]["who"])
print(choose_mode(flags), end=" -> ")
flags.set("answer.enabled", False, "oncall", "kill switch drill")
print(choose_mode(flags))

t= 10 min  v2 share   5%  v2 errors  2/32   proceed  
t= 20 min  v2 share  25%  v2 errors  9/132  proceed  
t= 30 min  v2 share  50%  v2 errors 50/287  rollback error rate 50/287 (low end 0.135) vs baseline 6/313
[(0.0, 0.05), (0.05, 0.25), (0.25, 0.5), (0.5, 0.0)] by rollout-bot
full -> unavailable

At 5% the new version had 2 errors in 32 requests and at 25% it had 9 in 132 (6.8% against 1.3%), but the low end of the interval stayed inside the margin, so the gate proceeded; at 50% it had 50 in 287 and the gate stopped. A load-dependent defect can pass the early stages, so later stages need the gate too; a stricter margin catches it sooner and stops more good releases. Every change of share is logged with who and why. Flags also holds the kill switch: with answer.enabled off, choose_mode returns unavailable, a clear temporary failure with no deploy. Use a flag to roll back a prompt or config; a model, dependency or application needs a redeploy.

Write a deployment rollback plan

A rollback plan is data a machine can check. For each unit (application, dependency, model, prompt, config) it names the current and previous exact versions, the step, an owner, a time budget and a test that must pass afterwards. The checker refuses latest as a target, a prompt rollback that needs a build, a missing unit, and a verify test that does not exist. Never upgrade a dependency and roll back in one release, and keep the previous image and model id as long as you might need them.

import dataclasses

from evidencedesk.serve import rollback

plan = rollback.load_plan()
print([u.kind for u in plan], rollback.problems(plan))
broken = [dataclasses.replace(u, previous="latest") if u.kind == "model" else dataclasses.replace(u, needs_deploy=True) if u.kind == "prompt" else u for u in plan if u.kind != "config"]
for problem in rollback.problems(broken):
    print("-", problem)

['application', 'dependency', 'model', 'prompt', 'config'] []
- no rollback step for config
- model: previous version 'latest' is not an exact, different version
- prompt: should be a flag change, not a deploy (a rollback must not wait for a build)

Plan capacity and cost

workers_needed applies Little's law per worker, runs each at 60% of its tighter limit, adds a spare and reports the load with one worker lost; monthly_cost separates compute (paid busy or idle) from tokens (which follow traffic).

from evidencedesk.serve.capacity import monthly_cost, workers_needed

need = workers_needed(40, 0.015, 0.8, slots=8)  # 40 requests/s at peak, 60% target use, one spare worker
print(need)
print(monthly_cost(need, 20_000_000, 1200))  # 20 million requests a month, 1,200 tokens each

Plan(peak_rps=40, workers=8, slot_limit_rps=78.5, cpu_limit_rps=533.3, utilisation_at_peak=0.509, utilisation_one_down=0.582)
{'compute': 292.0, 'tokens': 48000.0, 'total': 48292.0, 'per_request': 0.002415}

Eight workers for 40 requests per second at 8 slots each, 51% used at peak and 58% with one lost. With 64 slots one core would be the limit and 2 workers would do. In these invented numbers tokens cost about 160 times the compute, so a cache hit or a shorter prompt saves more than tuning workers.

Mini-lab: deployment readiness check and a game day

The readiness check runs the offline checks of this and the last page, then the tests.

python -m evidencedesk.serve.readiness

READINESS (offline)
[ok  ] acceptance list        20 criteria, each with a check
[ok  ] container files        no failing rule
[ok  ] smoke test             7 steps pass, no key involved
[ok  ] environments           dev, staging, production pass their rules
[ok  ] timeout chain          each layer gives up before the one above
[ok  ] load at planned peak   4 req/s: shed 3.3%, p95 1.18s
[ok  ] release gate           offline evaluation gate says review
[ok  ] game-day replay        gated canary exposed 451, ungated rolling 4714
[ok  ] rollback plan          5 release units, each with a previous version and a check
RESULT: READY

python -m pytest kit/tests/test_scale_*.py | tail -n 1 | sed 's/ in [0-9.]*s//'

55 passed

Run the game day as a team. One person edits data/serve/gameday.json (a new defect, a different slope, a cost regression) without telling the others; the rest replay it with the gate on, read the rollback reasons, and use the rollback plan to decide what to flip first. Hand in the readiness output, the replay table, the secrets checklist (E01 to E03) and the checked rollback plan. This cannot show that an image builds, or that real traffic and a real orchestrator behave like the simulation.

Where this shows up at work

Aiko in Osaka is on call for an assistant used by 400 support agents. After a provider incident she adds a retry budget, a deadline header and an approved fallback; after a bad prompt release, a canary with a gate and a kill switch. The next incident is a fifteen-minute rollback touching 3% of users. (An illustrative case.)

Common mistakes

• An unbounded queue. It turns overload into slow, late, billed answers. Bound it, drop expired work, return 503 with a hint.
• Timeouts that do not nest. The outer layer gives up and the work goes on. Check the chain; pass the remaining time down.
• Trusting a 5% canary. A load-dependent defect passes it. Gate every stage; keep rollback one step away.
• Failing over to an untested model. It answers, so nobody notices the quality or privacy change. Gate it, check the data class, record who served.

Try it yourself

1. Easy. Revisit Budgets, Caching, and Usage Limits. A user may make 5 calls a day; three workers each keep their own UsageLimiter. Predict how many of 30 calls are allowed, then compare with a shared limiter.
2. Medium. Revisit Monitoring, Budgets, and Incidents (cost alerts). Make the game-day candidate add no errors but cost 1.8 times as much, and replay the canary. Predict where it stops and why.
3. Hard. Revisit Containerisation and Streaming Deployment (no waiting room). With 8 slots per worker, find the smallest fleet that sheds under 1% at 40 requests per second, then at a spike of 60. Compare with the planner's 8.

Answers

Easy. Three separate limiters allow 3 × 5 = 15; a shared one allows 5.

for shared in (False, True):
    c = Cluster(3, shared, ManualClock(0.0), per_user=Quota(5, 10**6))  # a limit of 5 calls a day, three workers
    print("shared" if shared else "per worker", "->", sum(c.ask("omar", f"q{i}")["status"] == 200 for i in range(30)), "of 30 allowed")

per worker -> 15 of 30 allowed
shared -> 5 of 30 allowed

Medium. The cost rule fires at the first stage with 30 requests, so the release stops at 5% after 31 requests.

from dataclasses import replace

costly = replace(sc, candidate=replace(sc.candidate, error_slope=0.0, cost=0.0018))  # v2: no new errors, but 1.8x the cost per request
r = run_rollout(costly, "canary")
for s in r.steps:
    print(f"v2 share {s.share:.0%}, {s.canary_n} v2 requests -> {s.action}: {'; '.join(s.reasons)}")

v2 share 5%, 31 v2 requests -> rollback: cost 0.0018 vs 0.001

Hard. Six workers hold the peak (0.1% shed) but not the spike (12%); the planner's eight survive both (0.7% at 60). The 60% target and the spare buy spike headroom. Real fleets need more than this one-queue model.

for rate in (40, 60):  # planned peak, then a spike of 1.5x
    print(rate, "req/s:", {n: f"{100 * (r := simulate(Profile(rate), Service(n, 8))).shed / r.offered:.1f}% shed" for n in (4, 5, 6, 8)})

40 req/s: {4: '14.4% shed', 5: '3.2% shed', 6: '0.1% shed', 8: '0.0% shed'}
60 req/s: {4: '38.2% shed', 5: '24.1% shed', 6: '12.2% shed', 8: '0.7% shed'}

Pass checklist:

• Easy: 15 and 5, with the reason (a counter per process).
• Medium: stops at 5% on cost, no error rule fired.
• Hard: 6 for the peak, 8 for the spike, and what the simulation leaves out.

Summary

• IO-bound model calls need concurrency (slots) first and processes second; workers share nothing, so shared limits and caches need shared state.
• Load-test with a fake client in virtual time: the saturation curve and Little's law show the ceiling before real traffic.
• Bound the queue, shed with a retry hint, nest the timeouts, pass the remaining deadline down.
• Retry with backoff, jitter and a budget, honour Retry-After, make repeated actions idempotent.
• Fail over only to a gated, data-approved provider and record who served; check environment settings with rules.
• Release by canary with metric gates, roll back by flag or plan, rehearse in a game day; simulations show mechanics only.

Next: Ingestion and Permission-Aware Retrieval turns to the data side of production: ingestion contracts, idempotent re-ingestion, versions and provenance, deletions, and permission checks that survive change.