Cost optimization

Cost optimization is choosing, deliberately, which of several ways to spend fewer tokens and less time fits a system, not applying all of them by default. Preserve the required outcome, automation, and human effort when comparing designs. A lower model bill is not a saving if it hands unwanted work back to the user. The decision worksheet classifies a candidate design; it does not establish the cheapest suitable solution. Possible levers include: a smaller model for questions that don’t need a large one (routing), less text in the prompt (context engineering), reusing a prefix instead of resending it (prompt caching), deferring work that doesn’t need an instant answer (batching), capping how long an answer or a loop is allowed to run, and caching a whole answer rather than only a prompt prefix.

This topic is not a level on the ladder; it applies at every level, the way ops does, and every lever below is one of the levers ops names in general. This page goes one level deeper into each one: what it actually saves, and what it risks.

This page is sourced, not measured: every figure below is a maker’s own published rate, read off their page on the date given, and no lever here has been pulled on this site’s own traffic and scored.

def _cost(model_id: str, tokens_in: int, tokens_out: int, prices: PriceTable) -> float | None:
    price = prices.get(model_id)
    if price is None:
        return None
    per_in, per_out = price
    return round(tokens_in / 1000 * per_in + tokens_out / 1000 * per_out, 6)

def load_price_table(path: Path) -> PriceTable:
    """A price table file: `{"<model_id>": {"in_per_1k": ..., "out_per_1k": ...}, ...}`, USD per
    1,000 tokens. The caller states where these numbers came from; nothing here checks that."""
    raw = json.loads(path.read_text(encoding="utf-8"))
    return {model_id: (float(entry["in_per_1k"]), float(entry["out_per_1k"])) for model_id, entry in raw.items()}

Failure modes

At each level

Each level has one lever that pays before any of the others. The list below is that lever, not a catalog; ops covers what each level costs to run.

• Conventional software: nothing is billed per token, so no lever below applies. Worth listing because it is the cheapest answer the worksheet can reach.
• Direct prompting: model choice, and nothing else. Nothing repeats yet, so there is nothing to cache; nothing is deferred, so there is nothing to batch.
• Added context: what you put in the window, because you pay for it on every single call. Cutting it and caching the part that cannot be cut are the same lever seen from two sides: see context engineering.
• Workflows: the step that runs on every question whether it needs to or not. Find it first, and only then ask whether it could be batched or skipped.
• Tool use: not a lever but a budgeting rule. The model decides whether a tool call happens, so the same question costs one call or two, and an estimate that assumes the cheaper case is wrong for some fraction of traffic you do not control.
• Agent loops: the ceiling, before anything else. A loop with no fixed length has no natural stopping cost, which is what a single agent’s max_steps exists for.
• Teams of Agents: seat assignment. Every lever multiplies by the number of agents, so the largest single saving is which model sits in which seat.
• Always-on agents: how often it wakes up. A system that runs whether or not there is anything to do spends most of its budget deciding there is nothing to do.

Practices

• Settle the level before touching a lever. The worksheet decides whether the system is one level too high, which is worth more than every lever on this page put together.
• Divide by successful tasks, never by calls. A lever that wins on the wrong denominator is not a saving, and the two numbers move in opposite directions often enough to matter.
• Give each lever its own condition and check it on a schedule: a cache needs a hit rate, a batch queue needs nobody waiting, a router needs a real share of easy questions.
• Keep prices out of the code. A dated table supplied by the caller goes stale visibly; a hard-coded number goes stale silently and reads as a claim about a real price.
• Enforce the levers once, at the gateway, rather than reimplementing caching and budgets in every application that calls a model.

Run it

Try it

1. Use it

   Pick a task you currently do with a single, most-capable chat model. Using the worksheet's own questions, decide whether a cheaper level or a smaller model would still pass: then actually try it once and compare.

2. Build it

   Run python -m examples.ops --demo from the repo root and read the RAG row's tokens_in (1,850). Using the cache-write and cache-read multipliers quoted above, work out by hand how many reads inside a five-minute window it takes before caching that prefix has paid for its own write.

3. Either lane

   Pick one technique's Cost and latency strip elsewhere on this site. Name the one lever from this page that would cut its cost the most, and the one condition (named in this page's own failure modes) that would have to hold for that lever to actually pay off.