Monitoring, Budgets, and Incidents

Kwame in Accra ships a new prompt for the support assistant on a Tuesday morning. Nothing crashes. By lunch, one reply in three is a failed answer, and nobody knows since when: the only dashboard shows requests per minute, and that line is flat. (An illustrative case.) The traces from Structured Logging and Tracing record what happened to each request. They do not say that something is wrong, how wrong, who to wake, or what to switch off.

This page turns traces and logs into numbers, objectives, alerts and a response. Everything runs on seeded synthetic traffic and a fake clock, so each number is repeatable. Every threshold is illustrative: copy the mechanism, then choose your numbers from your own data.

What you will learn

• How to turn traces into a small metrics dataset, with rates and percentiles that refuse to answer on too little data.
• How to write SLOs and error budgets for quality, latency, cost and safety, and alert rules that do not flap.
• How to detect drift and regressions, and what their false-alarm rates are.
• How to bound retries, degrade safely, enforce soft and hard budgets, spot abuse and sample replies for human review.
• How to set retention and access, and run an incident from first alert to postmortem.

What is new, and what is reused

No existing file changes. evidencedesk/observe/ gains metrics, traffic, slo, alerts, drift, resilience, budgets, abuse, review, retention, incident and incident_lab; data/observe/ gains seven small JSON files (SLOs, alert rules, budgets, severity levels, runbook, retention, incident script); five test files are named test_observe_monitor_*.py. Reused: Lab, EventLog and percentile from the previous page; HitStats and the UTC-day rule of UsageLimiter from Budgets, Caching, and Usage Limits; RetryPolicy from Adapters, Configuration, and Error Handling; the hypothetical price table of Cost and Latency Fundamentals; status_of from Safe Failure and Security Regression Tests; kappa from Evaluation Design and Judge Calibration.

Measure cache hits and misses

A hit rate is honest only with the right denominator. A request that never reached the cache (a retrieval refusal) or may not use it (a summary) is not a miss. HitStats keeps hits, misses (eligible lookups that found nothing) and bypassed apart. metrics.from_trace flattens a finished trace into an Obs: status, code, latency, tokens, cost, cache state, best retrieval score, version, tenant; no text, no raw user id.

from evidencedesk.observe.lab import Lab
from evidencedesk.observe.metrics import from_trace, summarise

Q = "How long do I have to return a bike?"
lab = Lab()
for q in (Q, Q, "What is the capital of Australia?"):
    lab.ask(q)
seen = [from_trace(t, tenant="tenant-a") for t in lab.tracer.traces()]
for o in seen:
    print(o.request_id, o.status, o.code, o.cache, o.tokens_in, o.tokens_out, o.cost, o.top_score)
s = summarise(seen, min_n=1)
print("hit rate:", s.hit_rate, "| lookups:", s.cache.eligible, "| p95 with 3 requests:", summarise(seen).p95_s)

req-001 answered ok miss 326 39 0.001446 0.56
req-002 answered ok hit 0 0 0.0 0.56
req-003 refused no_evidence none 0 0 0.0 0.0
hit rate: 0.5 | lookups: 2 | p95 with 3 requests: None

The second request is a free hit; the refusal never touched the cache, so it is not counted. With three requests the 95th percentile is None: every rate and percentile in a Summary does this below min_n (20 here), because a percentile of three numbers is one number.

Build a small metrics dataset

The lab has too few requests to practise on, so traffic.make_day() generates a seeded day: 240 minutes at about six requests a minute from about 30 users in three tenants, a bad release at minute 150 and a scraper burst at minute 60. It stands in for production, which is burstier and stranger. buckets gives one summary per time bucket, empty buckets included, because a missing bucket is itself a signal. Counters only grow; a rate is the difference of two snapshots, never a reset.

from evidencedesk.observe import metrics, traffic

rows = traffic.make_day()
t0 = rows[0].ts


def hm(ts):
    return f"{int(ts % 86400 // 3600):02d}:{int(ts % 3600 // 60):02d}"


print(len(rows), "requests | users:", len({o.user for o in rows}), "| versions:", sorted({o.version for o in rows}))
print("start    n  error%  p95_s  cost/req(milli)  hit%  naive%")
for t, b in metrics.buckets(rows, t0, t0 + 4 * 3600, 1800):
    print(f"{hm(t)} {b.n:4d} {b.error_rate * 100:6.1f} {b.p95_s:6.2f} {b.cost_per_request * 1000:15.3f} {b.hit_rate * 100:5.1f} {b.cache.naive_rate * 100:6.1f}")

1580 requests | users: 31 | versions: ['v1', 'v2']
start    n  error%  p95_s  cost/req(milli)  hit%  naive%
09:00  180    4.4   1.31           0.944  31.4   27.0
09:30  186    9.7   1.33           0.958  29.3   27.1
10:00  313    2.9   1.31           0.899  17.1   15.4
10:30  182    5.0   1.31           1.074  24.1   22.0
11:00  180    6.7   1.29           1.080  24.2   21.8
11:30  179   25.1   1.70           1.527  19.9   18.2
12:00  179    3.4   1.28           1.084  24.8   22.8
12:30  180    4.4   1.30           1.030  24.5   22.9

This table is the dashboard dataset. The 11:30 bucket stands out in errors, p95 and cost. The last column is the wrong denominator: it understates the hit rate by about 1.6 to 4.4 points because summaries bypass the cache. The 10:00 bucket holds the scraper's 130 extra requests; the 09:30 error bump (18 in 186) is chance.

Tool note (as of 2026-10-12): The OpenTelemetry GenAI conventions list a metrics page beside the span pages, and the whole set has status Development, so names can change. This page computes its own metrics from spans and logs and calls no metrics SDK. If you export metrics later, copy names from the current page and check the current documentation.

Define SLIs, SLOs and error budgets

An SLI is a measured share of good requests; an SLO is the share you promise over a window; the error budget is what the promise leaves (at 95 percent, 5 requests in 100 may be bad). Burn rate is bad requests divided by budget: 1.0 spends it exactly at the end of the window, 14 spends it fourteen times too fast. An LLM feature needs four dimensions, because it can fail in four ways while the others look fine. Each SLI is a proxy; none says "the answer was right".

from evidencedesk.observe import slo

for b in slo.report(slo.load_slos(), rows):
    print(f"{b.slo:<16} n={b.n} bad={b.bad:<4} allowed={b.allowed:<5} sli={b.sli} budget left={b.remaining:+.2f} burn={b.burn_rate}")

valid-replies    n=1580 bad=115  allowed=79.0  sli=0.9272 budget left=-0.46 burn=1.46
fast-replies     n=1580 bad=44   allowed=79.0  sli=0.9722 budget left=+0.44 burn=0.56
cheap-replies    n=1580 bad=77   allowed=31.6  sli=0.9513 budget left=-1.44 burn=2.44
no-safety-flags  n=1580 bad=114  allowed=7.9   sli=0.9278 budget left=-13.43 burn=14.43

Quality and cost are overspent: the bad release. Safety is overspent fourteen times, and that is one user: the scraper put 140 requests, half blocked, into a budget that allows 8 in the day. A per-feature number cannot say who; the abuse section does. Latency has budget left because v2 replies were slower but mostly within the limit.

Create operational alerts from thresholds

A rule names a metric over a trailing window, a threshold to fire and a different one to clear, how long a breach must last, and how few requests make it silent. One rule as stored in alert_rules.json:

{"name": "error-rate-high", "metric": "error_rate", "op": ">", "fire": 0.10, "clear": 0.05, "for_min": 10, "window_min": 15, "min_n": 30, "severity": "SEV2"}

A for-duration ignores a spike that ends before anyone could act. Hysteresis (fire 0.10, clear 0.05) stops a metric hovering at the line from paging again and again. De-duplication keeps one open alert per rule key: one notification per change, a reminder at an interval. A minimum sample keeps three requests from raising an alarm. The state machine inside Engine.tick:

# excerpt of kit/evidencedesk/observe/alerts.py (Engine.tick)
            st = self.state[r.name]
            if st != "firing" and r.breached(v):
                self.since.setdefault(r.name, now)
                if now - self.since[r.name] >= r.for_min * 60:
                    self.state[r.name] = "firing"
                    self.last_note[r.name] = now
                    out.append(AlertEvent(now, r.name, "fired", v, r.severity))
                else:
                    self.state[r.name] = "pending"
            elif st == "firing" and r.recovered(v):
                self.state[r.name] = "ok"
                self.since.pop(r.name, None)
                out.append(AlertEvent(now, r.name, "resolved", v, r.severity))
            elif st == "pending" and not r.breached(v):

Replay the day, every five fake minutes:

from evidencedesk.observe import alerts

engine = alerts.run(alerts.load_rules(), rows, t0, t0 + 4 * 3600)
for e in engine.events:
    print(hm(e.ts), f"{e.rule:<22} {e.kind:<9} value={e.value:.4f} {e.severity}")
print("evaluations skipped for too little data:", engine.skipped)

11:45 error-rate-high        fired     value=0.3182 SEV2
11:55 cost-per-request-high  fired     value=0.0015 SEV3
12:15 error-rate-high        resolved  value=0.0330 SEV2
12:25 cost-per-request-high  resolved  value=0.0011 SEV3
evaluations skipped for too little data: 1

The release began at 11:30 and the page came at 11:45: 15 minutes to detect. The cost alert fired after the fix because its window is 30 minutes, and resolution lags too. That is the price of stable alerts: longer windows and for-durations remove noise and add delay. Now the same error rule on traffic that sits right at the threshold:

rule = next(r for r in alerts.load_rules() if r.name == "error-rate-high")
edge = traffic.stationary(600, 3, traffic.Profile(codes={"ok": 0.9, "invalid_json": 0.1}))  # error rate sits right at the threshold
for name, r in [("naive (no for, no hysteresis)", rule.naive()), ("for 10 min + hysteresis", rule)]:
    eng = alerts.run([r], edge, 0, 6000)
    print(f"{name:<30} notifications: {len(eng.events)}  fired: {sum(e.kind == 'fired' for e in eng.events)}")

naive (no for, no hysteresis)  notifications: 5  fired: 3
for 10 min + hysteresis        notifications: 2  fired: 1

Five notifications for one condition, against two.

Detect drift and regressions

Drift means inputs or outputs look different from a baseline; a regression means they got worse. Neither implies the other. drift.compare tests five signals: topic mix (chi-square), retrieval score distribution (two-sample Kolmogorov-Smirnov), refusal and error rates (two-proportion z-test) and cost per request (relative change). Baseline: everything before the release. Current: its 22 minutes.

from evidencedesk.observe import drift

base = [o for o in rows if o.ts < t0 + 150 * 60]
bad = [o for o in rows if t0 + 150 * 60 <= o.ts < t0 + 172 * 60]
for s in drift.compare(base, bad):
    print(f"{s.name:<17} baseline={s.baseline:<8} current={s.current:<8} stat={s.stat:<7} limit={s.limit:<7} {'FLAG' if s.flagged else ''}")

input_mix         baseline=0.0      current=0.0707   stat=3.37    limit=13.28   
score_dist        baseline=0.532    current=0.532    stat=0.0753  limit=0.1509  
refusal_rate      baseline=0.1152   current=0.0992   stat=-0.54   limit=3.0     
error_rate        baseline=0.0537   current=0.3053   stat=9.9     limit=3.0     FLAG
cost_per_request  baseline=0.000979 current=0.001698 stat=0.734   limit=0.25    FLAG

Errors and cost move; the input mix and retrieval scores do not. So users and retrieval did not change: something after retrieval did. That is evidence, not proof. The question every drift monitor must answer is how often it fires when nothing changed. Draw two windows from the same distribution, 100 times per size:

def same(n, seed):
    return traffic.stationary(n, seed)


for size in (30, 100, 400):
    naive = sum(drift.tv_distance([o.topic for o in same(size, 2 * t)], [o.topic for o in same(size, 2 * t + 1)]) > 0.15 for t in range(100))
    print(f"windows of {size:>3}: fixed mix distance flags {naive}%, all five signals: {drift.false_positive_rate(same, size, 100)}")

windows of  30: fixed mix distance flags 75%, all five signals: {'cost_per_request': 0.12, 'score_dist': 0.01, 'any': 0.13}
windows of 100: fixed mix distance flags 18%, all five signals: {'cost_per_request': 0.01, 'input_mix': 0.03, 'any': 0.04}
windows of 400: fixed mix distance flags 0%, all five signals: {'input_mix': 0.01, 'refusal_rate': 0.01, 'any': 0.02}

A fixed threshold on mix distance flags 75 percent of identical windows of 30 requests, because small samples wobble; the chi-square test, which knows sample size, does not. Yet five signals add up: 13 percent of identical pairs of 30 raise some flag (mostly cost, which is skewed), 2 percent at 400. So: use big windows, treat a flag as a question for a person, and require it to persist. A harmless change, such as a campaign about gift cards, flags the mix and breaks nothing (a test checks that), and a stale baseline silently absorbs a slow decline.

Monitor model and prompt changes

Every Obs carries the version in force (prompt, model, corpus). Comparing versions is the cheapest regression monitor and needs no tuned threshold:

for version, s in metrics.group_by(rows, lambda o: o.version).items():
    print(version, f"n={s.n} error={s.error_rate} p95={s.p95_s} cost/req={s.cost_per_request} tokens_out/req={s.tokens_out / s.n:.0f}")

v1 n=1448 error=0.0511 p95=1.309 cost/req=0.001002 tokens_out/req=34
v2 n=132 error=0.3106 p95=1.708 cost/req=0.00169 tokens_out/req=94

The new prompt costs 1.7 times as much per request because replies are almost three times longer, and fails six times as often. Put the version in every span and log line, change it through a flag, send a new version to a small share first, and compare per version rather than per day. A version that wins offline and loses online is a finding about the evaluation set from Evaluation Design and Judge Calibration.

Use bounded retries and backoff

A retry fixes a brief fault and worsens a long one. resilience.storm sends 12 new requests a second to a provider that answers up to 20 calls a second and none for 30 seconds, with your RetryPolicy. Three clients: retry at once, capped exponential backoff with full jitter, and backoff plus a retry budget (retries at most 10 percent of first tries).

from evidencedesk.observe import resilience as rz

for name, policy, budget in [("retry at once", rz.NAIVE, None), ("backoff + full jitter", rz.BACKOFF, None), ("backoff + retry budget", rz.BACKOFF, 0.1)]:
    st = rz.storm(policy, retry_budget=budget)
    print(f"{name:<24} calls={st.attempts:<5} peak/s={st.peak:<3} answered={st.served:<4} gave up={st.gave_up:<4} overloaded seconds={st.overloaded_seconds}")
box = rz.Outbox()
for _ in range(3):
    box.send("ak-demo", "ticket for order KC-1001")
print("side effect calls:", box.calls, "| tickets created:", len(box.sent))

retry at once            calls=2188  peak/s=48  answered=756  gave up=324  overloaded seconds=33
backoff + full jitter    calls=1814  peak/s=38  answered=746  gave up=334  overloaded seconds=32
backoff + retry budget   calls=1130  peak/s=24  answered=722  gave up=358  overloaded seconds=1
side effect calls: 3 | tickets created: 1

Retrying at once sends 2188 calls with a peak of 48 a second into a service that handles 20, and the provider stays overloaded for 33 seconds, three more than the outage. Jitter lowers the peak, not enough; the retry budget halves the calls and ends the overload. The trade is visible: it answers 34 fewer requests. Bounded retries choose whose requests to drop. Honour a server's wait hint, retry only calls that are safe to repeat, and count every attempt in the ledger. A side effect is safe to repeat only with an idempotency key chosen by the application, not the model (the last line: three calls, one ticket).

Provider note (as of 2026-10-12): A cloud provider's engineering blog describes capped exponential backoff, min(cap, base * 2 ** attempt), and three jitter variants (Full, Equal, Decorrelated); jitter is there because retries otherwise cluster. HTTP Retry-After is a delay in seconds or a date, sent with 503 and allowed with 429; MDN says client support is still inconsistent. A server may also simply refuse the connection. The retry budget and load-shedding advice are our own. Check the current documentation.

Design safe degradation

When the model is down, costly or suspect, the feature should shrink, not break. resilience.MODES runs from full through shorter_output, cheaper_model and search_only (the best passages as sources, no model call) to unavailable, a clear temporary failure with a retry hint. A circuit breaker opens after three failures in a row so callers stop waiting on a dead dependency, allows one trial after a pause, and closes on success. A flag with an audit trail is the kill switch; changing it needs no deploy. One function decides, strongest reason first:

# excerpt of kit/evidencedesk/observe/resilience.py
def choose_mode(flags: Flags, breaker: Breaker | None = None, budget_mode: str = "") -> str:
    """Strongest reason wins: kill switch, then an operator's fixed mode, then an open breaker, then the budget's degrade mode."""
    if not flags.get("answer.enabled"):
        return "unavailable"
    if (m := flags.get("answer.mode")) != "auto":
        return str(m)
    if breaker is not None and not breaker.allow():
        return "search_only"
    return budget_mode or "full"

Four timeouts, a recovery, then the switch:

lab = Lab()
flags, breaker = rz.Flags(lab.clock), rz.Breaker(lab.clock, threshold=3, reset_s=30.0)


def step(label, model_mode="ok"):
    mode = rz.choose_mode(flags, breaker)
    if mode == "full":
        reply = lab.ask(Q, model_mode)
        breaker.record(reply.status != "failed")
        outcome = reply.code
    elif mode == "search_only":
        outcome = "sources: " + ", ".join(p["id"] for p in rz.search_only(lab.retriever, Q)["passages"][:2])
    else:
        outcome = rz.unavailable()["code"]
    print(f"{label:<12} mode={mode:<12} breaker={breaker.state:<9} {outcome}")


for i in range(4):
    step(f"call {i + 1}", "timeout")
lab.clock.advance(31)
step("after 31 s", "ok")
flags.set("answer.enabled", False, "on-call", "kill switch")
step("kill switch")
print([(h["flag"], h["to"], h["who"]) for h in flags.history])

call 1       mode=full         breaker=closed    timeout
call 2       mode=full         breaker=closed    timeout
call 3       mode=full         breaker=open      timeout
call 4       mode=search_only  breaker=open      sources: kc-returns-v3:c0, kc-warranty-v2:c0
after 31 s   mode=full         breaker=closed    ok
kill switch  mode=unavailable  breaker=closed    temporarily_unavailable
[('answer.enabled', False, 'on-call')]

The breaker opened on the third timeout, so call 4 never touched the model. After 31 fake seconds one trial succeeded and it closed. The kill switch overrides everything and the history says who and why. Search-only is honest but a worse experience: tell the user, and test the degraded path before the day you need it.

Track budget consumption

UsageLimiter counted calls and tokens per user and overall. A product also needs a feature scope and a gentler step before the wall. BudgetKeeper keeps daily counters per user, tenant and feature in calls and credits (an invented unit). Below the soft limit (80 percent here) requests pass; between soft and hard they are served in a cheaper mode, stronger as use grows; at the hard limit they get the stable code budget_exceeded with scope and reset time. The tightest scope decides, and a call that would cross the line is refused before it is made.

from evidencedesk.observe import budgets
from evidencedesk.usage import ManualClock

keeper = budgets.BudgetKeeper(ManualClock(20000 * 86400.0), budgets.Policy.load())
last = None
for call in range(1, 46):
    v = keeper.check("u1", "tenant-a", "answer", 0.001)
    if (v.action, v.mode) != last:
        print(f"call {call:>2}: {v.action:<8} {v.mode:<15} {v.code} scope={v.scope or '-'} used={v.used}")
        last = (v.action, v.mode)
    if v.action != "block":
        keeper.record("u1", "tenant-a", "answer", 0.001)
print("tenant-b meanwhile:", keeper.check("u2", "tenant-b", "answer").action, "| notices:", keeper.notices)

call  1: allow                    ok scope=- used=0.03
call 32: degrade  shorter_output  budget_soft scope=user used=0.8
call 35: degrade  cheaper_model   budget_soft scope=user used=0.88
call 38: degrade  search_only     budget_soft scope=user used=0.95
call 41: block                    budget_exceeded scope=user used=1.02
tenant-b meanwhile: allow | notices: ['user:u1 crossed the soft limit']

Degraded from call 32, blocked from call 41; another tenant is untouched. A notice is written once at the soft limit so a person hears before users do. Limits in application code are a guard, not a bill: set the provider's own spend limits as well.

Anomaly and abuse signals

Abuse looks like ordinary traffic from one place. abuse.signals reports bursts (most requests in any 60 seconds), suspected injection flags, a user's blocked share and a user's share of spend; shares need a minimum request count.

from evidencedesk.observe import abuse

for s in abuse.signals(rows):
    print(s.line)
print("largest request shares:", abuse.top_share(rows)[:2])

u-0e6bb8a5 burst value=35.0 limit=15.0 n=140
u-0e6bb8a5 injection_flags value=77.0 limit=5.0 n=140
u-0e6bb8a5 blocked_share value=0.486 limit=0.3 n=140
largest request shares: [('u-0e6bb8a5', 0.089), ('u-3fb4ad39', 0.039)]

One pseudonym trips three signals; nobody else trips any. Signals rank who a person should look at; they must not ban anyone. A school sending thirty students to one tool looks like a burst. Tie them to a graduated response (slow down, degrade, block).

Quality sampling and the human review queue

No threshold sees a fluent wrong answer; only a person reading replies does, and human time is a budget. review.select fills the queue in two lanes: a random lane, the only unbiased view of quality, and a targeted lane (failures, flags, answers despite a low retrieval score, abstentions, a new version), most important first. Without the reserved random lane, failures fill the queue and the quality estimate is biased. Verdicts carry no text; reviewers read replies in the external content store with its short retention. Queue.agreement scores two reviewers with kappa (tested, not shown).

from evidencedesk.observe import review

items = review.select(rows, capacity=40, new_versions=frozenset({"v2"}))
by_id = {o.request_id: o for o in rows}
sampled = [by_id[i.request_id] for i in items if i.reason == "sampled"]
good = sum(o.truth == "good" for o in sampled)
print("queue:", {r: sum(i.reason == r for i in items) for r in sorted({i.reason for i in items})})
print(f"random lane: {good}/{len(sampled)} good, 95% interval {review.wilson(good, len(sampled))}")
for n in (10, 20, 100, 400):
    print(f"if 80% of {n} reviewed items are good, the 95% interval is {review.wilson(round(0.8 * n), n)}")

queue: {'blocked': 5, 'failed': 11, 'flagged': 4, 'sampled': 20}
random lane: 16/20 good, 95% interval (0.584, 0.919)
if 80% of 10 reviewed items are good, the 95% interval is (0.49, 0.943)
if 80% of 20 reviewed items are good, the 95% interval is (0.584, 0.919)
if 80% of 100 reviewed items are good, the 95% interval is (0.711, 0.867)
if 80% of 400 reviewed items are good, the 95% interval is (0.758, 0.836)

Sixteen of 20 sampled replies were good, but the interval runs from 58 to 92 percent. A quality change of a few points needs hundreds of reviewed items, so a 5 percent sample cannot see a 22-minute regression. That is why loud signals and per-version comparison come first, and the sample catches the slow, quiet loss. Each reviewed failure becomes a case for the evaluation set.

Plan log retention and access

Observability data is data. retention.json states, per class (metrics, logs, traces, content, review labels, postmortems), what it holds, days kept, which roles read it, how it is deleted and whether it is sensitive. The numbers are examples; your rules come from your legal, security and customer commitments. The checks catch plan mistakes, not legal ones:

from evidencedesk.observe import retention

plan = retention.load_plan()
print("plan problems:", retention.plan_problems(plan))
print({c.name: (c.days, "/".join(c.access)) for c in plan if c.name in ("logs", "content")})
broken = [retention.DataClass("logs", "events", 30, ("everyone",), "", False), retention.DataClass("content", "text", 90, ("reviewers",), "purge", True)]
for p in retention.plan_problems(broken):
    print(" -", p)
day = 86400.0
kept, gone = retention.purge([("c1", "content", 0.0), ("c2", "content", 9 * day), ("l1", "logs", 0.0), ("x1", "scratch", 9 * day)], 10 * day, plan)
print("kept:", [r[0] for r in kept], "| deleted:", gone, "| product may read content:", retention.may_read(plan, "product", "content"))

plan problems: []
{'logs': (30, 'engineering'), 'content': (3, 'reviewers/incident-lead')}
 - metrics: no rule
 - traces: no rule
 - logs: needs days > 0 and a way to delete
 - logs: access must name roles, not everyone
 - content: sensitive data kept 90 days (illustrative cap: 7)
 - content must be kept for less time than metadata
kept: ['c2', 'l1'] | deleted: {'content': 1, 'scratch': 1} | product may read content: False

The checker found a missing rule, a missing deletion method, "everyone" as audience, sensitive data kept too long and content outliving metadata. purge also deletes what has no class: no rule, no retention. A user's deletion request must reach the content store too.

Write an incident triage checklist

When the page arrives, name the kind of incident, because each has a different first move. The checklist is code, so it answers from metrics, and security comes first because it may need action the others do not.

from evidencedesk.observe import incident

normal = traffic.stationary(300, 1)
cases = {"model keeps timing out": {"ok": 0.6, "timeout": 0.25, "server": 0.1}, "nothing is retrieved": {"ok": 0.5, "no_evidence": 0.4},
         "blocked requests spike": {"ok": 0.4, "not_allowed": 0.5}, "quiet window": {"ok": 0.9}}
for label, codes in cases.items():
    t = incident.triage(traffic.stationary(300, 2, traffic.Profile(codes=codes)), normal)
    print(f"{label:<24} -> {t.category:<10} matches={t.matches}")
levels = incident.load_levels()
print("SEV2:", levels["SEV2"]["impact"], "| respond within", levels["SEV2"]["respond_min"], "min")
print("regression runbook:", incident.load_runbook()["regression"]["steps"][1])

model keeps timing out   -> provider   matches=['provider']
nothing is retrieved     -> retrieval  matches=['retrieval']
blocked requests spike   -> security   matches=['security']
quiet window             -> unknown    matches=[]
SEV2: The feature is down or failing for many users | respond within 15 min
regression runbook: Set prompt.version back to the previous value

Severity follows user impact, not interest in the cause: here 25 percent bad is SEV2 and 5 percent SEV3 (illustrative), and a suspected leak is SEV1. The runbook is data with an owner, ordered steps and a "verify" line: stop the harm first, find the cause second, say how you will know it worked. Several kinds can be true at once; the checklist returns all matches and leads with the first.

Mini-lab: replay a bad release

incident_lab runs the day end to end: traffic, the alert engine, an acknowledgement two minutes after the page, triage, the rollback as a flag change seven minutes after the page, recovery, a timeline merged from log events and trace facts, and a postmortem generated from them. It refuses to run if the script and the response disagree about when the rollback lands. The log is the previous page's EventLog: codes and numbers, no text, no raw user id.

python -m evidencedesk.observe.incident_lab

alerts: 11:45 error-rate-high fired; 11:55 cost-per-request-high fired; 12:15 error-rate-high resolved; 12:25 cost-per-request-high resolved
triage: regression | validation failure share 0.322 (baseline 0.028); new versions: ['v2']
# Postmortem INC-001: Release v2 raised invalid replies

Severity SEV2. Category regression.

## Summary
Between 11:30 and 11:52 41 of 132 requests failed (31%); replies that parsed but were wrong are not counted here. 52% of the day's quality budget was spent.

## Detection and response
- Started 11:30, detected 11:45 (time to detect 15 min).
- Mitigated 11:52 (time to mitigate 7 min), resolved 12:15.

## Timeline
- 11:30 [log] deploy.started {"change": "prompt text and reply template", "version": "v2"}
- 11:30 [trace] first request on version v2
- 11:30 [trace] first failed request on v2 (truncated)
- 11:45 [log] alert.fired {"rule": "error-rate-high", "severity": "SEV2", "value": 0.3222}
- 11:47 [log] oncall.ack {"rule": "error-rate-high"}
- 11:50 [trace] last failed request on v2 (invalid_json)
- 11:52 [log] flag.changed {"flag": "prompt.version", "reason": "rollback", "to": "v1"}
- 11:55 [log] alert.fired {"rule": "cost-per-request-high", "severity": "SEV3", "value": 0.0016}
- 12:15 [log] alert.resolved {"rule": "error-rate-high", "severity": "SEV2", "value": 0.0333}
- 12:25 [log] alert.resolved {"rule": "cost-per-request-high", "severity": "SEV3", "value": 0.0011}

## Cause
The v2 prompt and reply template made the model leave the JSON format and run longer. No check on validation failures per version ran before release.

## What helped
- The error-rate alert fired 15 minutes after the release
- Rollback was a flag change: no build, no deploy

## What did not
- Nothing compared validation failures per version before release
- The cost alert fired after the fix, because its window is 30 minutes

## Actions
| What | Owner (role) | Due |
|---|---|---|
| Ship prompt changes to 5% of requests first and compare per version | release owner | 2 weeks |
| Alert on the validation-failure rate per version | app on-call | 1 week |
| Add the failed v2 cases to the regression set | evaluation owner | 1 week |

The postmortem names roles, never people, and blameless_problems lints for blame words and names. Its counts leave out replies that parsed but were wrong; the review queue is for those. The 15-minute delay and the late cost alert result from the thresholds chosen here, not from a law of incidents.

python -m pytest kit/tests/test_observe_monitor_{metrics,alerts,budgets,review,incident}.py | tail -n 1 | sed 's/ in [0-9.]*s//'

48 passed

The tests prove the code paths. They cannot show that real traffic has this shape or that your thresholds suit your users.

Where this shows up at work

Priya in Pune joins a team whose assistant had one dashboard and no alerts (an illustrative case). In two weeks she adds per-version error and cost panels, an error alert with a 10-minute for-duration, a kill-switch flag and a one-page runbook. The first real page, a provider timeout storm, takes six minutes from alert to search-only.

Common mistakes

• Alerting on a raw threshold with no sample size. One failure in three requests is 33 percent. Require a minimum count and a for-duration.
• One number for fire and clear. The alert flaps. Add hysteresis and count notifications per incident.
• Treating a drift flag as a defect. It is a question; small windows and many signals raise false alarms. Use big windows and persistence.
• Retrying without limits or keys. It multiplies load and duplicates side effects. Cap attempts, add a retry budget and an idempotency key.
• Reviewing only failures. The quality estimate is biased. Keep a random lane and report the interval, not just the share.

Try it yourself

1. Easy. Revisit Cost and Latency Fundamentals. Compute cost per request for each version of the synthetic day. Predict first whether v2 costs more than 1.5 times v1.
2. Medium. Revisit Budgets, Caching, and Usage Limits. Give each user 10 calls a day with the same 80 percent soft limit. Predict the call numbers where the mode changes, then check.
3. Hard. Revisit Structured Logging and Tracing. Because every observation has a version, compare the first requests on v2 with the v1 baseline every ten requests from the 30th. Find when error_rate is first flagged and compare it with the window alert.

Answers

Easy. About 1.7 times: v2 replies are nearly three times longer, and output tokens cost more than input tokens in the price table.

from evidencedesk.observe import metrics

per = metrics.group_by(rows, lambda o: o.version)
print({v: round(s.cost_per_request * 1000, 3) for v, s in per.items()}, "| ratio v2/v1:", round(per["v2"].cost_per_request / per["v1"].cost_per_request, 2))

{'v1': 1.002, 'v2': 1.69} | ratio v2/v1: 1.69

Medium. Soft at 80 percent of 10 is call 8; each of the three modes lasts one call; call 11 is refused.

from dataclasses import replace

small = replace(budgets.Policy.load(), user=budgets.Limit(10, 1.0))
k2 = budgets.BudgetKeeper(ManualClock(20000 * 86400.0), small)
marks, last = [], None
for call in range(1, 14):
    v = k2.check("u", "t", "answer")
    if (v.action, v.mode) != last:
        marks.append((call, v.action, v.mode))
        last = (v.action, v.mode)
    if v.action != "block":
        k2.record("u", "t", "answer", 0.0)
print(marks)

[(1, 'allow', ''), (8, 'degrade', 'shorter_output'), (9, 'degrade', 'cheaper_model'), (10, 'degrade', 'search_only'), (11, 'block', '')]

Hard. A per-version comparison needs only 30 requests and compares like with like, so it flags at 11:34, eleven minutes before the window alert. It needs a version label on every request.

v1 = [o for o in rows if o.version == "v1" and o.ts < t0 + 150 * 60]
v2 = [o for o in rows if o.version == "v2"]
for k in range(30, len(v2) + 1, 10):
    if "error_rate" in drift.flagged(drift.compare(v1, v2[:k])):
        print(f"flagged after {k} requests on v2, at {hm(v2[k - 1].ts)}; the window alert fired at 11:45")
        break

flagged after 30 requests on v2, at 11:34; the window alert fired at 11:45

Pass checklist:

• Easy: 1.002 and 1.69 milli-credits per request, ratio 1.69, and you named reply length as the cause.
• Medium: transitions at calls 8, 9, 10 and 11 (shorter output, cheaper model, search-only, block).
• Hard: flagged after 30 requests at 11:34, and you said what it needs (a version label) and cannot see (a fault that is not tied to a version).

Summary

• Flatten traces into observations, count with the right denominators, refuse rates and percentiles below a minimum sample, and use buckets (empty ones too) and per-version groups as the dashboard dataset.
• Write SLOs for quality, latency, cost and safety, track the budget and burn rate, and build alerts with a for-duration, hysteresis, de-duplication and a minimum sample; longer windows trade noise for delay.
• Drift flags are questions with measurable false-alarm rates; compare versions, use big windows, and keep a random review lane because no metric sees a fluent wrong answer.
• Bound retries with backoff, a retry budget and idempotency keys; degrade in steps with a breaker and a kill switch; enforce soft and hard budgets per user, tenant and feature.
• Plan retention and access as data, triage by evidence, follow a runbook and write blameless postmortems from the timeline.

Next: Threat Modelling and Injection Testing maps the assets, actors and trust boundaries of the whole assistant and tests direct and indirect prompt injection with safe local attacks.