What it means
Model APIs bill by token, with output typically costing several times more than input. Total cost is therefore driven by how much context you send and how much text you generate, on every single call.
The usual surprise is that input dominates. A retrieval system stuffing several documents into context, or a chat application resending an entire conversation each turn, sends far more tokens than the user's visible message. Reasoning models invert this by generating large amounts of intermediate output before answering.
Two mechanisms materially change the arithmetic. Prompt caching discounts repeated context heavily, which matters enormously for a fixed system prompt or a document queried repeatedly. Batch processing discounts non-urgent work. Both are routinely left unused.
Per-token prices have fallen steeply and continue to, which is why cost models built a year ago tend to be pessimistic.
Why it matters
This is what determines whether an AI feature survives contact with scale. The failure pattern is consistent: a feature is prototyped with a flagship model on small volume, and the unit economics only become visible once real usage arrives.
In practice
Estimate tokens per request including context, then multiply by realistic volume before committing. Route easy work to cheaper models, enable caching for repeated context, and instrument token usage per feature so you know where the money goes.