Prompt Caching and Cost: Stable Prefixes, Usage Data and Estimates

Many AI requests repeat the same long beginning: the same instructions, the same examples, the same document, followed by a short new question. Providers can reuse the work they did on an identical beginning, which is called prompt caching. It can lower cost and delay, but only if you build prompts so that the beginning really is identical, and only if you check the usage data to see what happened. This page shows how to order a prompt for caching, what caching is not, how to read hits from usage data, and how to run a controlled experiment on settings.

The code uses the standard library and promptlab. The cache here is a toy provider: an exact-prefix cache simulation, a model of the idea and not a real service. It reports hits through the cached_input_tokens field of Usage. All prices are example prices, not real: made-up units so you can practise the arithmetic. Real providers differ in rules, minimums, lifetimes and prices, so look them up.

What you will learn

• How to separate a stable prefix from variable input, and why order matters.
• What prompt caching is and is not: not memory, not retrieval, not fine-tuning.
• How to measure cache hits only from usage data that reports them.
• How to reason about latency, cost, retention and pricing before production.
• How to compare quality before and after a settings change in a controlled experiment.

Stable prefixes versus variable inputs

Split every prompt into two parts. The stable prefix is what is the same on every call: standing instructions, output rules, examples, a reference document. The variable input is what changes: this user's question, today's record. Caching rewards a prompt that puts all stable material first, in the same order, byte for byte, and the variable input last. This agrees with the advice you met in Long-Context Evaluation: Chunking, Position Tests and Lost Details (long material first, question last), and it is also what providers recommend for caching.

Weak prompt:

Today is 2026-10-12. The customer is Mateo. Question: what is the refund window?
You answer questions about the shop policy. Use only the policy. Be brief.
### POLICY
(several pages of policy text)

Better prompt:

You answer questions about the shop policy. Use only the policy. Be brief.
### POLICY
(several pages of policy text)
### REQUEST
Today is 2026-10-12. The customer is Mateo.
Question: what is the refund window?

What changed: the date and the customer name, which change on every call, moved to the end. In the weak version the first line differs from call to call, so nothing after it can match an earlier call.

Caching is reuse of repeated context

Caching reuses the provider's computation for a prefix it has already processed. It does not change what the model knows or how it answers; the answer should be the same quality with or without a hit. It is an optimisation of cost and delay, and a hit is never guaranteed.

Caching is not conversation memory

Memory means the assistant carries facts from earlier chats into a new one. A cache stores nothing you can ask about. A cache hit only means that the beginning of this request equals the beginning of a recent request, so the provider did not have to process it again. You still send the whole prompt every time, and the model sees only what you send.

Caching is not retrieval or fine-tuning

They solve different problems and can be combined. Retrieval and fine-tuning belong to the AI Engineer path.

Prefix ordering across providers

Provider note (as of 2026-10-12): OpenAI's documentation says cache reuse requires the entire rendered prefix to match, and that reuse does not guarantee a hit. Anthropic's documentation says the cache covers tools, then system, then messages, up to a breakpoint; hits need 100% identical segments, and a change at one level invalidates the later levels. Google's documentation advises putting large, common content at the beginning and sending requests with a similar prefix close together in time. Caching can be implicit (the provider decides) or explicit (you mark a breakpoint or manage a cache object), and availability differs by provider and model. Each provider also has a model-specific minimum prompt length; below it nothing is cached, and Anthropic says that happens without an error. Check the current documentation for your model.

The common rule is exact-prefix reuse. A cache hit needs an identical beginning, so one changed early character spoils everything after it. The first code block builds a toy cached model with a minimum length, block-sized reuse and a lifetime, and runs it on a stable-first layout and a variable-first layout, with the same questions.

import re
from promptlab import Client, estimate_tokens

POLICY = {"refund window": "14 days", "warranty": "24 months", "shipping fee": "6.50", "return label": "free",
          "support hours": "08:00-18:00", "gift wrap": "2.00", "price match": "7 days", "repair time": "10 days"}
INSTRUCTIONS = "You answer questions about the shop policy below. Use only the policy. Be brief.\n"
DOC = "### POLICY\n" + "\n".join(f"- {k}: {v}. " + "Applies to all orders placed on the web shop. " * 3 for k, v in POLICY.items())

class ToyCachedModel:
    """Stand-in provider with an exact-prefix cache. It models the idea; it is not a real service."""
    BLOCK, MINIMUM, TTL = 16, 64, 5          # toy numbers: reuse in blocks, a minimum length, entries expire after 5 ticks

    def __init__(self):
        self.seen, self.clock = {}, 0         # prompt text -> tick when last used

    def __call__(self, messages, max_tokens=200, idle=1, **settings):
        self.clock += idle
        self.seen = {p: t for p, t in self.seen.items() if self.clock - t <= self.TTL}
        prompt = "\n".join(m["content"] for m in messages)
        best = 0
        for old in self.seen:                 # longest shared beginning with any recent prompt
            n = 0
            while n < min(len(old), len(prompt)) and old[n] == prompt[n]:
                n += 1
            best = max(best, n)
        cached = best // 4 // self.BLOCK * self.BLOCK
        cached = cached if cached >= self.MINIMUM else 0
        self.seen[prompt] = self.clock
        topic = re.search(r"### QUESTION\nWhat is the (.*?)\?", prompt).group(1)
        text = f"The policy covers this topic in one line, and the line applies to all orders. Answer: {POLICY[topic]}"
        return {"text": text[: max_tokens * 4], "cached_input_tokens": cached}

QUESTIONS = [f"What is the {k}?" for k in POLICY]
def layout(kind, q):
    if kind == "stable_first":
        return [{"role": "system", "content": INSTRUCTIONS + DOC}, {"role": "user", "content": "### QUESTION\n" + q}]
    return [{"role": "user", "content": "### QUESTION\n" + q + "\n" + INSTRUCTIONS + DOC}]

client = Client(ToyCachedModel())
for i, q in enumerate(QUESTIONS[:3], 1):
    u = client.complete(layout("stable_first", q)).usage
    print(f"stable-first call {i}: input {u.input_tokens}, cached {u.cached_input_tokens}")
client = Client(ToyCachedModel())
for i, q in enumerate(QUESTIONS[:3], 1):
    u = client.complete(layout("variable_first", q)).usage
    print(f"variable-first call {i}: input {u.input_tokens}, cached {u.cached_input_tokens}")

stable-first call 1: input 358, cached 0
stable-first call 2: input 356, cached 352
stable-first call 3: input 358, cached 352
variable-first call 1: input 358, cached 0
variable-first call 2: input 357, cached 0
variable-first call 3: input 358, cached 0

The first call can never hit, because nothing came before it. In the stable-first layout the next calls reuse most of the prompt. In the variable-first layout the question comes first, the beginnings differ at once, and nothing is reused although the later text is identical. Same words, different order, different result.

One early change spoils the rest

Teams often break a cache without noticing: a timestamp in the first line, a changed example, a reordered list. Test your own template by editing one place and watching the usage data.

def cached_after(edit):
    model = Client(ToyCachedModel())
    model.complete(layout("stable_first", QUESTIONS[0]))
    messages = layout("stable_first", QUESTIONS[1])
    messages[0] = {"role": "system", "content": edit(messages[0]["content"])}
    return model.complete(messages).usage.cached_input_tokens

print("no edit           :", cached_after(lambda s: s))
print("edit at the end   :", cached_after(lambda s: s[:-12] + "CHANGED-END."))
print("edit at the start :", cached_after(lambda s: s.replace("You answer", "You respond", 1)))

no edit           : 352
edit at the end   : 336
edit at the start : 0

An edit near the end keeps most of the earlier reuse; an edit at the start removes all of it. Put the parts that change least first, and the parts that change most last.

Measure cache hits only where usage data exposes them

You cannot see a cache; you can only read what the provider reports. Treat "the usage data shows cached tokens" as the only evidence of a hit. If a provider does not report a cache field, "zero" means unknown, not miss, and a latency drop alone proves nothing, because load and output length also change latency.

Provider note (as of 2026-10-12): OpenAI reports cached reads in the usage details. Anthropic reports cache_creation_input_tokens, cache_read_input_tokens and input_tokens, which sum to the total input. Google reports a cached-token count in the usage metadata; the caching page names usage.total_cached_tokens for its Interactions API, and I could not confirm the field name for its other APIs. If both cache counts are zero, nothing was cached. Check the current documentation.

def toy_no_usage(messages, **settings):
    return "ok"                               # a provider that reports no cache field at all

def cache_status(response, exposed):
    if not exposed:
        return "unknown (no cache field reported)"
    return "hit" if response.usage.cached_input_tokens else "miss"

cached_client, plain_client = Client(ToyCachedModel()), Client(toy_no_usage)
for _ in range(2):
    r1 = cached_client.complete(layout("stable_first", QUESTIONS[0]))
    r2 = plain_client.complete(layout("stable_first", QUESTIONS[0]))
print("toy cache  :", cache_status(r1, exposed=True))
print("no field   :", cache_status(r2, exposed=False), "| raw value:", r2.usage.cached_input_tokens)

toy cache  : hit
no field   : unknown (no cache field reported) | raw value: 0

The raw value of the second line is 0 but the status is "unknown". In a log, store the full usage object for every call, plus your own wall-clock time, because latency is not a field in these providers' usage data.

Cache effects on latency and cost

A hit can reduce the delay before the first output and the price of the repeated input. It does not reduce output cost, and it does not help a prompt that is below the provider's minimum or is never repeated. Some providers' documentation describes a price shape like this: the first use (a cache write) can cost more than normal input, and repeated reads cost a small fraction. Other pages do not describe the structure, so read each price sheet. The toy uses example prices, not real, per million tokens, for ordinary input, cached input and output; it ignores any write premium. In the toy model the latency is a fake number, so do not read it: with a real provider, measure latency yourself over many calls.

Retention and pricing checks before production

A cache entry does not last forever. Lifetimes are minutes to hours depending on provider and settings; a longer lifetime can have its own price. If your traffic is occasional, the entry may expire between calls and you pay full price. The toy has a short lifetime, shown by configuration C below. Before production, check on the provider's current pages: the minimum cacheable length for your model, the retention and how it is refreshed, the price for writes and reads, whether caching is implicit or needs a marker, and whether your data policy allows it. Then compute a cost estimate from your own traffic, not from a demo.

Compare quality before and after settings changes

Any change, whether a layout, a limit or a model setting, needs the same test: same cases, one change at a time, the same score. Caching should leave quality unchanged and cost lower; a different output cap can change both. The harness below runs four configurations over the same eight questions and prints right answers, tokens, hit share and an example cost for 1,000 repeats of the set (example units).

PRICE = {"input": 1.00, "cached": 0.10, "output": 4.00}     # example prices per million tokens, not real

def run_experiment(kind, max_tokens=200, idle=1):
    client = Client(ToyCachedModel())
    ok = cached = total_in = out = 0
    for q in QUESTIONS:
        r = client.complete(layout(kind, q), max_tokens=max_tokens, idle=idle)
        ok += POLICY[q[12:-1]] in r.text
        cached += r.usage.cached_input_tokens
        total_in += r.usage.input_tokens
        out += r.usage.output_tokens
    cost = ((total_in - cached) * PRICE["input"] + cached * PRICE["cached"] + out * PRICE["output"]) / 1e6
    return ok, total_in, cached, cost

print(f"{'config':24} {'right':>5} {'input':>6} {'cached':>6} {'hit':>4} {'example cost x1000':>18}")
for name, kind, mt, idle in [("A stable first", "stable_first", 200, 1), ("B variable first", "variable_first", 200, 1),
                             ("C stable, slow traffic", "stable_first", 200, 10), ("D stable, small cap", "stable_first", 20, 1)]:
    ok, tin, ca, cost = run_experiment(kind, mt, idle)
    print(f"{name:24} {ok:>3}/8 {tin:>6} {ca:>6} {ca / tin:4.0%} {cost * 1000:18.2f}")

config                   right  input cached  hit example cost x1000
A stable first             8/8   2859   2464  86%               1.37
B variable first           8/8   2862      0   0%               3.59
C stable, slow traffic     8/8   2859      0   0%               3.59
D stable, small cap        0/8   2859   2464  86%               1.28

Read it as an experiment. A is the baseline. B has the same quality but higher cost, because the layout wastes the cache. C has the same layout as A but calls that are too far apart, so entries expire: the cache saved nothing. D is cheapest-looking per call and wrong on every question, because the output cap cut off the answer; a cheaper run that fails the quality check is not an improvement. Decide with both columns together. The toy is deterministic, so one run each is enough here. With a real provider, repeat each configuration several times, because quality and latency vary from run to run (see Sampling and Reasoning Controls: Temperature, Top-p, Effort and Repeatability).

Where this shows up at work

Support assistants send the same policy text with every question. Document-review tools ask many questions about one contract. Evaluation runs send one long instruction block with hundreds of test cases. All of them gain from a stable prefix, and all of them should confirm the gain in the usage data of their own provider before they plan a budget around it. Long-Context Evaluation: Chunking, Position Tests and Lost Details showed how to estimate size before a run; this page adds what the run should cost when the prefix repeats.

Common mistakes

• Putting the date, user name or request id first. Every call then has a different beginning. Put variable parts last.
• Believing a cache remembers the chat. It reuses computation on an identical prefix. You still send everything.
• Claiming a hit without usage data. Read the cached-token fields; where none exist, say "unknown".
• Planning a budget from a demo. Minimums, lifetimes and prices differ. Check the price sheet and your real traffic pattern.
• Accepting a cheaper setting without a quality check. Compare the same cases before and after.

Try it yourself

Run the page's blocks in order in one Python folder with promptlab.py beside them.

1. Easy. A prompt starts with "Request id 8841, 2026-10-12 09:15." and then 3,000 tokens of fixed rules. Why will exact-prefix caching rarely help, and how do you fix it?
2. Medium. Using cached_after, edit the middle of the system text (for example the gift wrap line) and report the cached tokens. Where does the cached amount fall, compared with the start and end edits?
3. Hard. Run a controlled experiment: add configuration E, stable-first with max_tokens=200 and idle=3, and a decision rule that accepts a configuration only if it gets all eight right and its example cost is not higher than A's. Report which of B, C, D and E pass.

Answers

Easy. The first line changes on every call, so the beginning never matches an earlier request and the 3,000 tokens after it are not reused. Move the request id and time to the end, after the fixed rules. Pass checklist: names the changing first line; moves variable data last; mentions checking the usage data afterwards.

Medium.

print("edit in the middle:", cached_after(lambda s: s.replace("gift wrap: 2.00", "gift wrap: 3.00", 1)))

edit in the middle: 224

The cached amount lies between the two earlier results: it covers the text before the edit and loses what follows. Pass checklist: edit position explained as "the matching stops at the first difference"; the three results compared; no real-provider claim from the toy.

Hard.

base = run_experiment("stable_first")
for name, args in [("B", ("variable_first", 200, 1)), ("C", ("stable_first", 200, 10)),
                   ("D", ("stable_first", 20, 1)), ("E", ("stable_first", 200, 3))]:
    ok, tin, ca, cost = run_experiment(*args)
    print(name, "accept" if ok == 8 and cost <= base[3] else "reject", f"(right {ok}/8, hit {ca / tin:.0%})")

B reject (right 8/8, hit 0%)
C reject (right 8/8, hit 0%)
D reject (right 0/8, hit 86%)
E accept (right 8/8, hit 86%)

A is the baseline, so it is not in the loop. B and C cost more, and D fails the quality rule. E is accepted because calls three ticks apart still fall within the toy lifetime, so its cost equals A's. Pass checklist: quality and cost are both in the rule; the baseline is compared fairly; the answer says E's result depends on the toy's invented lifetime and a real provider's retention must be looked up.

Summary

• Put stable material first and variable input last, in the same order every time.
• Prompt caching reuses an identical prefix; it is not memory, retrieval or fine-tuning, and it does not change the answer.
• Claim a hit only from usage data that reports cached tokens; a missing field means unknown.
• Check minimum length, retention and price shape in the current documentation before you plan a budget.
• Compare quality and cost together, on the same cases, changing one thing at a time.

Next: Python Prompt Experiments: Test Cases, Runners, JSONL Logs and Mock Mode, where you turn these one-off experiments into reusable runners with logs.