The cost of inference: budgeting AI like infrastructure

Every AI programme reaches the same meeting. Finance asks what one request costs. Engineering answers in tokens per second and monthly aggregates, and the meeting ends without a number anyone can plan against.

The gap is not financial literacy. It is that tokens are a unit of consumption, not a unit of value. Nobody buys tokens. They buy a resolved ticket, an extracted invoice, a drafted contract clause, a validated claim. Until the cost model is expressed in those units, the system is being monitored rather than budgeted.

Start from the billable unit, not the API call#

Pick the unit the business already counts. For a support assistant it is a resolved conversation. For a document pipeline it is a processed document. Then measure the fan-out between that unit and model calls — because it is never 1.0.

Between one unit and the invoice sit query rewrites, a reranking pass, two or three tool calls, a guardrail check, a schema-repair retry, and a final summarisation. A conversation that feels like one question to the user is commonly six to twelve model calls in the trace.

Two numbers make the model usable: cost per unit at p50 and at p95. The median pays the monthly bill. The 95th percentile sizes the blast radius when a customer pastes a forty-page PDF into the chat box and an agent decides to read all of it.

type Rates = {
  in: number;          // USD per million input tokens
  out: number;         // USD per million output tokens
  cacheRead: number;   // USD per million tokens served from prefix cache
  cacheWrite: number;  // USD per million tokens written to prefix cache
};

// Illustrative tiers. Substitute your provider's current published rates.
const RATES = {
  frontier: { in: 3.0, out: 15.0, cacheRead: 0.3, cacheWrite: 3.75 },
  small: { in: 0.8, out: 4.0, cacheRead: 0.08, cacheWrite: 1.0 },
} satisfies Record<string, Rates>;

type Tier = keyof typeof RATES;

export type Usage = {
  tier: Tier;
  inputTokens: number;
  cacheReadTokens: number;
  cacheWriteTokens: number;
  outputTokens: number;
};

const PER_MTOK = 1_000_000;

export function callCost(u: Usage): number {
  const r = RATES[u.tier];
  return (
    (u.inputTokens * r.in +
      u.cacheReadTokens * r.cacheRead +
      u.cacheWriteTokens * r.cacheWrite +
      u.outputTokens * r.out) /
    PER_MTOK
  );
}

/** The number finance actually asked for. */
export function unitCost(trace: Usage[], resolvedUnits: number): number {
  const total = trace.reduce((sum, call) => sum + callCost(call), 0);
  return total / Math.max(resolvedUnits, 1);
}

Note what the function does not take: throughput. Unit cost is independent of traffic volume, which is why the first optimisation is never “scale it down”. Volume changes the bill. Only the unit model changes the business case.

The money is in the input, not the output#

Engineers instinctively optimise the part they can see: the answer. But a typical assistant call carries a 20:1 ratio of input to output tokens, and at frontier rates that still leaves input as the larger line item. In the baseline below it is 82 per cent of the spend.

What is actually filling the window is rarely audited:

• The system prompt and policy text, which accretes a paragraph per incident and is never pruned.
• Tool schemas. A registry of forty tools with verbose JSON Schema descriptions runs to several thousand tokens on every single call, including the calls that use no tools at all.
• Few-shot exemplars, often kept long after a model upgrade made them unnecessary.
• Retrieved context — the most common source of waste, because top-k is set once and never revisited.
• Conversation history, replayed in full on every turn unless something compacts it.

A worked model#

Take a support assistant answering from a product knowledge base. Baseline: a 9,000-token static prefix (system prompt plus tool schemas), 5,000 tokens of retrieved passages and history, 600 tokens of answer. Rates are the illustrative tiers above.

The last row is the one that matters. Four rounds of optimisation produced an 83 per cent reduction on paper, and the retry and repair traffic handed 4 per cent of it back. Cost models that stop at row four are the reason forecasts miss.

Prompt caching rewards prompts that hold still#

Prefix caching stores the key-value tensors for a prompt prefix so the next request skips prefill over that span. Providers typically charge a premium to write the cache — on the order of 1.25× the base input rate — and roughly a tenth of the base rate to read it, with a time-to-live measured in minutes.

That yields a break-even worth memorising. Writing costs 0.25 extra; each hit saves 0.9. A single hit therefore more than repays the write, so the cache is ahead from the second request against the same prefix — provided that request lands inside the TTL.

Which means cache economics are a function of traffic per distinct prefix, not total traffic. A service doing 500 requests a minute across 400 tenant-specific system prompts concentrates that traffic in a handful of tenants — the long tail of prefixes never sees a second request inside the TTL and pays the write premium every time.

1. Order the prompt by volatility: policy and tool schemas first, then long-lived reference material, then retrieved context, then the user turn.
2. Never put a timestamp, request ID, or user name at the top of the system prompt. One variable token invalidates every cached token beneath it.
3. Keep tenant-specific text below the shared prefix so the expensive span stays common across tenants.
4. Instrument cache hit ratio per prefix, not per service. A healthy aggregate routinely hides a prefix that never hits.
5. Treat a prompt edit as a deployment: it invalidates the cache fleet-wide and the bill spikes for one TTL window.

Routing and cascades#

A cascade sends every request to a small model first and escalates on a confidence gate. The arithmetic is unforgiving in a useful way: the cascade pays the small tier on every request and the large tier on escalations, so its cost is c_small + e × c_large where e is the escalation rate.

It beats always-large while e < 1 − c_small / c_large. With the tiers above that threshold sits near 73 per cent — comfortable headroom. Cascades usually fail not on the ratio but on the gate.

The gate must be cheap and calibrated. Sequence log-probability, a trained classifier over the draft answer, or a retrieval-confidence signal all work. Using a second model as judge does not: the judge call frequently costs more than the escalation it was meant to avoid.

Routing is the simpler sibling and often the better first move. Classify intent once, then send the class to the tier it needs. Extraction against a fixed schema rarely needs a frontier model. Multi-step reasoning over a contract does.

Batch what nobody is waiting for#

Asynchronous batch endpoints commonly trade a multi-hour SLA for roughly half the price. Embedding backfills, nightly summarisation, eval suite runs, catalogue enrichment and re-indexing all belong there. Anything inside a user’s request path does not.

The architectural move is to split interactive and deferred work at design time rather than discovering the split during a cost review. A queue, an idempotency key and a results table are cheaper than the discount is large.

On self-hosted inference the same idea appears as continuous batching. With vLLM or SGLang, GPU cost is per second of occupancy, so the metric to optimise is tokens per second per GPU, not price per token. Paged attention, FP8 or AWQ quantisation and speculative decoding all move that number.

When fine-tuning beats a bigger prompt#

Fine-tuning is a capital expense that buys a marginal saving. Distilling a 3,000-token few-shot block into LoRA weights saves roughly $0.0024 per call at small-tier input rates. Against a realistic one-off of data curation, two or three training runs, an eval harness and regression testing, break-even lands somewhere past three million calls.

So the honest rule is that fine-tuning for cost alone pays only at high, stable volume. Fine-tune for behaviour you cannot reliably prompt: strict output format, a house tone, a domain vocabulary, a classification boundary that few-shot examples keep getting wrong.

And state the tradeoff plainly. A fine-tune pins the system to one base model version. Every base model upgrade re-opens the training cost, and knowledge baked into weights goes stale silently where retrieval would have been corrected by a re-index.

The retry tax#

Retries are invisible in the design document and entirely visible in the invoice. Four sources account for most of them: schema violations that trigger a re-ask, guardrail rejections, tool errors that send the agent round the loop again, and client timeouts.

Agent loops deserve their own line item. Cost grows with the number of turns, and because each turn appends its result to the context, the cost of each turn grows too. Spend is quadratic in loop length unless something compacts the transcript.

Budget ceilings you can actually enforce#

A ceiling checked after the fact is a report, not a control. Enforce it by reserving the worst case before the call — prompt tokens plus the maximum output the request is allowed to generate — then reconciling against actual usage afterwards.

export class BudgetExceeded extends Error {}

export type Budget = {
  ceilingUsd: number;
  spentUsd: number;
  hops: number;
  maxHops: number;
};

export function newBudget(ceilingUsd: number, maxHops = 8): Budget {
  return { ceilingUsd, spentUsd: 0, hops: 0, maxHops };
}

/** Worst-case cost of a call, known before it is issued. */
export function reserve(
  tier: Tier,
  promptTokens: number,
  maxOutputTokens: number,
): number {
  const r = RATES[tier];
  return (promptTokens * r.in + maxOutputTokens * r.out) / PER_MTOK;
}

export function admit(b: Budget, cost: number): void {
  if (b.hops + 1 > b.maxHops) {
    throw new BudgetExceeded("hop limit reached");
  }
  if (b.spentUsd + cost > b.ceilingUsd) {
    throw new BudgetExceeded("request budget ceiling reached");
  }
}

/** Charge actual usage once the provider reports it. */
export function settle(b: Budget, actual: Usage): Budget {
  return { ...b, spentUsd: b.spentUsd + callCost(actual), hops: b.hops + 1 };
}

What happens when admission fails is a product decision, and it should be an explicit one: downshift the tier, truncate retrieval, shorten the answer, or fail closed with a message the user can act on. Whichever it is, emit a metric. Silent degradation is how a cost control becomes a quality incident.

Set the ceiling per request and per tenant per day. The per-request ceiling stops one pathological input. The per-tenant ceiling stops one customer’s integration loop from consuming a month of margin over a weekend.

Cost and latency are the same conversation#

Almost every lever above moves both numbers, because both are functions of tokens processed. Fewer input tokens means less prefill, which means a lower time to first token. A prefix cache hit skips prefill outright. Fewer output tokens is the strongest latency lever there is, since decoding is sequential.

Cascades are the interesting case: they lower median latency and raise the tail, because an escalated request pays for both tiers. Quote p50 and p95 together or the conversation is dishonest.

Batching is the one place the two genuinely diverge. Throughput improves, latency worsens. That is a trade to make deliberately per workload — never a global default.

What to instrument first#

• Per call: model tier, input tokens, cached-read tokens, cache-write tokens, output tokens, latency, finish reason.
• Per unit: total cost, call fan-out, and the trace ID that links them — cost belongs on the span, not in a separate ledger.
• Cache hit ratio broken down by prefix, alerting on a drop after any prompt deployment.
• Escalation rate for every cascade, and the gate’s precision against a labelled sample.
• Unit cost at p50 and p95, segmented by tenant and by intent class.
• Spend attributable to retries, guardrail rejections and timeouts, reported as a percentage of total.

OpenTelemetry spans carrying token attributes, aggregated into ClickHouse or whatever the platform already runs, are sufficient. The tooling is not the hard part. The discipline of treating an unattributed token the way you would treat an unattributed database query is.

Inference is infrastructure. It has a unit cost, a demand curve, a cache, a queue and a tail. Budget it that way and the finance meeting gets a number. Budget it as magic and it gets a forecast that will be wrong by the quarter.