Reasoning Trade-offs: Stepwise Prompts, Checks, Stopping Rules and Cost

Every extra step you add to a prompt (a plan, examples, a check, a second pass) can raise accuracy, and every one of them also adds cost, delay and new ways to fail. Good prompt design is a series of trade-offs that you settle with measurements, not with habits. This page continues the previous one: it compares plans with checklists, shows how to place constraints, examples and independent checks, adds stopping rules, and ends by measuring several approaches on the same cases.

You need Python 3.12 or newer and the standard library only. The blocks share variables, so type them in order. Every "model" in the code is a toy provider: a small deterministic stand-in with a planted slip, so the harness can be tested repeatably. Real models are less predictable, which is why you test them; no number printed by a toy says anything about a real model.

What you will learn

• When to use a fixed checklist, a model-generated plan, worked examples, or constraints placed before the task.
• How to split a long problem into bounded stages with tests, add an independent check for high-impact numbers, and list unresolved subproblems.
• How to set stopping conditions, and how to use reasoning settings only on models that support them.
• How to compare approaches by accuracy, wrong answers released, calls and rough size on one fixed set of cases.

The running problem: a bulk-order quote

A shop quotes notebooks. The rules are invented for this page. The code states them once and keeps a trusted reference function, truth, used only to score answers.

import re
from decimal import ROUND_HALF_UP, Decimal as D

RULES = ("Price 4.00 per notebook. Discount on goods: 5% from 100 notebooks, 12% from 500. "
         "Shipping 15.00, free when goods after discount reach 400.00. Tax 8% on goods after discount only.")
CASES = [20, 99, 100, 250, 500, 800]


def r2(x):
    return x.quantize(D("0.01"), ROUND_HALF_UP)


def truth(qty):
    pct = D("0.12") if qty >= 500 else D("0.05") if qty >= 100 else D(0)
    net = r2(D(4) * qty * (1 - pct))
    return r2(net + r2(net * D("0.08")) + (D(0) if net >= 400 else D(15)))


print("  ".join(f"{q}: {truth(q)}" for q in CASES))

20: 101.40  99: 442.68  100: 425.40  250: 1026.00  500: 1900.80  800: 3041.28

Model-generated plans or fixed checklists

Compare model-generated plans with fixed checklists. A fixed checklist is written by you once: the steps are known, reviewed and the same every time. A model-generated plan is proposed for each task: it adapts to unusual problems but can leave out a step or phrase it vaguely. Use the checklist for recurring tasks with known rules and costly errors. Use a plan for new or open-ended problems. In between, combine them: let the model propose a plan, and let code test that the required steps are present.

REQUIRED = {"discount": "discount", "shipping": "shipping", "tax": "goods only", "recheck": "recompute"}


def plan_gaps(plan, required=REQUIRED):
    text = " | ".join(plan).lower()
    return sorted(key for key, phrase in required.items() if phrase not in text)


toy_plan = ["Read the quantity", "Apply the volume discount", "Work out shipping from discounted goods",
            "Add tax", "Show the total"]  # stand-in for a plan written by a model
print(plan_gaps(toy_plan))

['recheck', 'tax']

The plan sounds sensible, but "Add tax" does not say that shipping is excluded, and nothing rechecks the total. The fixed list of required steps caught both. A reviewed checklist is the safety net; the plan is the flexibility.

Constraints before the solution

Specify constraints before asking for a solution. When the rules come after the question, the model may already be committed to an approach, and rules mentioned in passing are easy to miss. Put rules and the output format first, then the data, then the question last. Provider guides also advise putting long material first and the question at the end.

Weak prompt:

A customer wants 250 notebooks. What is the total? By the way, price is 4.00 each, 5% off from
100, 12% off from 500, shipping 15.00 unless goods reach 400.00, and tax is 8%.

Better prompt:

Rules:
1. Price: 4.00 per notebook.
2. Discount on goods: 5% from 100 notebooks, 12% from 500 notebooks.
3. Shipping: 15.00, free when goods after discount reach 400.00.
4. Tax: 8% on goods after discount only, never on shipping.
Output: one line each for goods, discount, goods after discount, shipping, tax, total (2 decimals).
Order: 250 notebooks.
Question: what is the total?

What changed: rules are numbered and complete (the weak version never says what tax applies to), the output shape is fixed, and the question comes last.

Worked examples for repetitive reasoning

Use worked examples for repetitive reasoning patterns. When the same multi-step pattern repeats, one or two complete examples show the layout and order of the steps better than a description. Pick examples that differ (one with a discount, one without), keep them in the same format as the output you want, and keep them out of your test cases.

Prompt:

Example 1. Order: 30 notebooks.
Goods 120.00. Discount 0%. Goods after discount 120.00. Shipping 15.00 (below 400.00).
Tax 9.60. Total 144.60.
Example 2. Order: 120 notebooks.
Goods 480.00. Discount 5%. Goods after discount 456.00. Shipping 0.00 (reaches 400.00).
Tax 36.48. Total 492.48.
Now do the same for: 250 notebooks.

An error in an example gets copied. Check the examples with code before you use them, and make sure no example is also a test case:

EXAMPLES = {30: "144.60", 120: "492.48"}
for qty, total in EXAMPLES.items():
    print(qty, total, "correct" if D(total) == truth(qty) else "WRONG: fix the example")
print("examples reused as test cases:", sorted(set(EXAMPLES) & set(CASES)) or "none")

30 144.60 correct
120 492.48 correct
examples reused as test cases: none

Bounded, testable stages

Break long problems into bounded, testable stages. A stage is bounded when it does one job, receives only what it needs and has a limit (here, one call). It is testable when its output can be checked on its own. Below, the quote is built in four stages (discount, goods after discount, shipping, total). Each stage is a toy model call with a range test. toy_direct is the one-shot alternative, with two typical slips planted: off-by-one tier limits and tax on shipping. toy_ship has one planted slip: it decides shipping from the goods before the discount.

USAGE = {"calls": 0, "chars": 0}


def call(model, prompt):
    reply = model(prompt)
    USAGE["calls"] += 1
    USAGE["chars"] += len(prompt) + len(reply)
    return reply


def field(prompt, name):
    return D(re.search(rf"^{name}: (\S+)", prompt, re.M).group(1))


def toy_direct(prompt):  # one-shot arithmetic with two planted slips
    qty = int(field(prompt, "Qty"))
    pct = D("0.12") if qty > 500 else D("0.05") if qty > 100 else D(0)
    net = r2(D(4) * qty * (1 - pct))
    return str(r2((net + (D(0) if net >= 400 else D(15))) * D("1.08")))


def toy_pct(prompt):
    qty = field(prompt, "Qty")
    return str(D("0.12") if qty >= 500 else D("0.05") if qty >= 100 else D(0))


def toy_net(prompt):
    return str(r2(field(prompt, "Goods before discount") * (1 - field(prompt, "Discount"))))


def toy_ship(prompt):  # planted slip: uses the goods before the discount
    return "0" if field(prompt, "Goods before discount") >= 400 else "15"


def toy_total(prompt):
    net = field(prompt, "Goods after discount")
    return str(r2(net + r2(net * D("0.08")) + field(prompt, "Shipping")))


def staged(qty):
    gross, head = D(4) * qty, f"{RULES}\nQty: {qty}\n"
    pct = D(call(toy_pct, head + "Reply with the discount only."))
    if pct not in (D(0), D("0.05"), D("0.12")):
        return None, "stage failed: discount"
    net = D(call(toy_net, head + f"Goods before discount: {gross}\nDiscount: {pct}\nReply with goods after discount."))
    if not D(0) <= net <= gross:
        return None, "stage failed: goods after discount"
    ship = D(call(toy_ship, head + f"Goods before discount: {gross}\nGoods after discount: {net}\nReply with shipping."))
    if ship not in (D(0), D(15)):
        return None, "stage failed: shipping"
    total = D(call(toy_total, head + f"Goods after discount: {net}\nShipping: {ship}\nReply with the total."))
    return (total, "ok") if total >= net else (None, "stage failed: total")


total, status = staged(100)
print(total, status)

410.40 ok

All four stage tests passed, yet the total is wrong. For 100 notebooks the goods are 400.00 before the discount, so the slip waived shipping, but after the discount they are 380.00 and shipping applies. Range tests catch absurd values (a discount of 50%, a negative total); they do not catch a plausible wrong one. That needs the next step.

Independent checks for high-impact calculations

Request independent checks for high-impact calculations. An independent check re-derives the result by a different route and compares. The best route for exact arithmetic is ordinary code, written and reviewed separately from the model's steps. Asking the same model to "double-check" in the same conversation is weaker, because it can repeat its own slip. When a number moves money, a mismatch should stop the item and send it to a person; it should never be silently corrected.

def independent_check(qty, total):
    return total == truth(qty)


total, status = staged(100)
print(total, status, "check passed" if independent_check(100, total) else f"MISMATCH, expected {truth(100)}: send to a person")

410.40 ok MISMATCH, expected 425.40: send to a person

Know what this check can and cannot do. It catches slips in arithmetic and in following the rules. It cannot catch a rule that was misunderstood the same way in both routes, so the rules themselves must be reviewed by a person who owns them.

Provider note (as of 2026-10-12): Anthropic's guide says an instruction such as "Before you finish, verify your answer against [criteria]" catches errors, but that its newest model verifies on its own, and older verification instructions can cause over-verification. Re-test such lines whenever the model changes. Check the current documentation.

Unresolved subproblems

Ask the model to identify unresolved subproblems. A model asked for a total will often fill a gap with a plausible guess. Give it an allowed alternative: a field that lists what it could not settle, and a rule that a non-empty list stops automatic release.

Prompt:

Return JSON with: total (or null if you cannot compute it) and unresolved (a list).
Do not guess. If something needed is neither in the rules nor in the order, add it to unresolved.
Order: 250 notebooks, invoice to a school.

Example reply (illustrative; yours will differ):

{"total": "1026.00", "unresolved": ["The order does not say whether the school is tax-exempt."]}

What to notice: the model computed the total under the stated rules and also flagged a fact the rules do not cover. The application, not the model, decides what happens next:

def can_release(record):
    return record.get("total") is not None and record.get("unresolved") == []


print(can_release({"total": "1026.00", "unresolved": []}))
print(can_release({"total": "1026.00", "unresolved": ["tax-exempt status unknown"]}))
print(can_release({"total": "1026.00"}))

True
False
False

A record without the field fails too, so a model that skips the list cannot pass by silence.

Stopping conditions for multi-step tasks

Set a clear stopping condition for multi-step tasks. Any loop (review, revise, review again) needs three exits: success (no open issues), a limit (a maximum number of rounds), and no progress (the same issues remain). Without them a loop can run until the budget is gone. The reviser here is a stand-in that fixes one fixable issue per round.

def refine(issues, fixable, max_rounds=3):
    for done in range(max_rounds + 1):
        if not issues:
            return f"done after {done} rounds"
        if done == max_rounds:
            return f"stop: round limit, hand over {issues}"
        fix = [i for i in issues if i in fixable][:1]
        if not fix:
            return f"stop: no progress after {done} rounds, hand over {issues}"
        issues = [i for i in issues if i != fix[0]]


print(refine(["units missing", "tone"], {"units missing", "tone"}))
print(refine(["a", "b", "c", "d", "e"], set("abcde")))
print(refine(["wrong rule"], set()))

done after 2 rounds
stop: round limit, hand over ['d', 'e']
stop: no progress after 0 rounds, hand over ['wrong rule']

Each run ends for a different, named reason, and the two failures hand the open issues to a person instead of looping or hiding them.

Reasoning modes only where supported

Compare reasoning modes only where the model supports them. A setting that one model accepts may be rejected, ignored or unnecessary on another. Keep a table of supported settings per target, send only those, and record what you dropped, so a missing setting never explains a quality difference unnoticed. The names here are neutral labels; the real names are in the note below.

SUPPORTED = {"target_a": {"effort"}, "target_b": {"effort", "temperature"}, "target_c": set()}


def apply_settings(requested, target):
    used = {k: v for k, v in requested.items() if k in SUPPORTED[target]}
    return used, sorted(set(requested) - set(used))


for target in sorted(SUPPORTED):
    print(target, apply_settings({"effort": "high", "temperature": 0.2}, target))

target_a ({'effort': 'high'}, ['temperature'])
target_b ({'effort': 'high', 'temperature': 0.2}, [])
target_c ({}, ['effort', 'temperature'])

Provider note (as of 2026-10-12): OpenAI documents reasoning.effort with levels that differ per model and calls it a tuning knob, not the primary way to recover quality; Anthropic documents adaptive thinking with an effort setting, and thinking can be always on for some models; Google documents thinking_level, which can be lowered to cut cost and latency. On all three, thinking tokens are billed as output and count toward the output limit, so leave headroom. Responses report usage including reasoning or thought tokens (OpenAI output_tokens_details.reasoning_tokens); latency is not a response field in these docs, so measure it on the client. Check the current documentation.

Measure accuracy, latency and cost on the same cases

Measure accuracy, latency and cost on the same task set. Three approaches on six cases: A, one-shot; B, four stages; C, four stages plus the independent check. Report more than accuracy: wrong answers released, escalations, model calls, a rough size, and calls in a row per case, which is a simple stand-in for waiting time. The rough size uses the rule characters / 4, which is an estimate only; with a real provider, use its usage fields. For real latency, wrap each call with time.perf_counter() on the client and store the elapsed time with the model identifier. Table columns: right is correct and released, wrong is wrong and released, sent is sent to a person, calls is the total of model calls, ~size is the rough size, and chain is calls in a row per case.

def evaluate(name, quote, check, cases):
    USAGE.update(calls=0, chars=0)
    right = wrong = sent = 0
    for qty in cases:
        total, _ = quote(qty)
        if total is None or (check and not independent_check(qty, total)):
            sent += 1
        elif independent_check(qty, total):
            right += 1
        else:
            wrong += 1
    row = (name, right, wrong, sent, USAGE["calls"], USAGE["chars"] // 4, USAGE["calls"] // len(cases))
    print(f"{row[0].ljust(16)}{row[1]:>7}{row[2]:>7}{row[3]:>7}{row[4]:>7}{row[5]:>8}{row[6]:>7}")


def one_shot(qty):
    return D(call(toy_direct, f"{RULES}\nQty: {qty}\nReply with the total.")), "ok"


def table(cases):
    print("approach".ljust(16) + "  right  wrong   sent  calls   ~size  chain")
    evaluate("A one-shot", one_shot, False, cases)
    evaluate("B staged", staged, False, cases)
    evaluate("C staged+check", staged, True, cases)


table(CASES)

approach          right  wrong   sent  calls   ~size  chain
A one-shot            2      4      0      6     322      1
B staged              5      1      0     24    1513      4
C staged+check        5      0      1     24    1513      4

Read the columns, not just the first one. A releases four wrong quotes. B releases one, which is the quote that costs money without anyone noticing. C releases none: that case went to a person, which is the point of the check. The price is visible too: A makes one call per case, B and C make four calls in a row, so they are slower and larger. If a wrong quote is cheap to correct, A or B may be enough; if it is not, C earns its cost. These numbers come from stand-ins, so only the method carries over: same cases, same scoring, one table.

Where this shows up at work

Quoting, invoicing, payroll checks and eligibility decisions all need a trade-off table like this before a model is allowed near money. Teams keep the case set under version control, rerun it for every prompt or model change, and review the "wrong released" column first, because it measures harm, while accuracy alone hides it.

Common mistakes

1. Trusting a plan because it reads well. Test it against a fixed list of required steps.
2. Asking the same model to check itself and calling that independent. Use code or another route that can disagree.
3. Judging by accuracy only. Add wrong answers released, escalations, calls, rough size and waiting time.
4. Looping with no exit. Set success, a round limit and a no-progress rule, and say who gets the leftovers.
5. Sending a setting to a model that rejects it. Keep a supported-settings table and log what was dropped.

Try it yourself

Use the code from this page; the blocks share one namespace. The exercises revisit Reasoning Task Design: Decomposition, Rationales and Intermediate Schemas and the case-set idea from Provider Differences and Portability: Reading Guides, Testing Across Models.

1. Easy. Run refine on the five-issue list with max_rounds=5. What is the stop reason, and what does it say about choosing the limit?
2. Medium. Add approach D, one_shot with the independent check, and print the table. What does D release, what does it escalate, and how does its cost compare with C?
3. Hard. Compare three approaches (A, B, C) to this business problem on a longer case list: add 105 and 106, which sit around the free-shipping boundary. Print the table, then write a two-sentence decision for a shop where a wrong quote costs a refund and a lost customer.

Answers

Easy.

print(refine(["a", "b", "c", "d", "e"], set("abcde"), max_rounds=5))

done after 5 rounds

Pass checklist: with five rounds the loop finishes; you saw that the limit decides whether the same job succeeds or is handed over, so the limit is a cost decision to make on purpose, not a default.

Medium.

evaluate("D one-shot+check", one_shot, True, CASES)

D one-shot+check      2      0      4      6     322      1

Pass checklist: D releases no wrong quote, but it sends many cases to a person because the one-shot slips are frequent; it uses one call per case, so it is cheaper than C and shifts the work to people. The right choice depends on what a person's review costs.

Hard.

table(CASES + [105, 106])

approach          right  wrong   sent  calls   ~size  chain
A one-shot            3      5      0      8     430      1
B staged              6      2      0     32    2018      4
C staged+check        6      0      2     32    2018      4

Pass checklist: case 105 is wrong in B (goods 399.00 after discount, so shipping applies) and is caught by C; the table has the same columns for all three approaches; the decision names the column it relies on. A strong answer: "Use C, because B still releases a wrong quote at the shipping boundary and a wrong quote costs more than four calls. Put the arithmetic in code for the next release, and keep the model for reading the order text."

Summary

• Use a reviewed checklist for known tasks and a model-generated plan for new ones, and test any plan for required steps.
• Put constraints and the output format before the data and the question. Use correct, varied worked examples that are not test cases.
• Split long problems into bounded stages with tests, add an independent check for high-impact numbers, and let the model list what is unresolved.
• Give every loop a success exit, a round limit and a no-progress exit. Use reasoning settings only where the model supports them.
• Compare approaches on the same cases with several columns, and read "wrong answers released" first.

Next: Prompt Chaining: Stages, Fixed Schemas and the promptlab Helper turns stages like these into a chain with fixed schemas and introduces the shared promptlab helper.