AI costs

Prompt caching and context trimming for long conversations

Updated 6 min readMarkdown
In short

A chat request sends the whole conversation again on every turn, so input cost grows with the square of the conversation's length. Prompt caching bills the repeated beginning of a prompt at a lower cache-read price. Context trimming leaves out earlier exchanges the latest message does not need. The two work against each other, because a gap in the history breaks the cache for everything after it. Trimming pays when the removed tokens at the price they were really billed at are worth more than the cache discount lost behind the gap plus the cost of selecting.

A chat request carries the whole conversation every time, so the tenth turn pays again for the nine before it. Two techniques lower that bill. Prompt caching makes the repeated beginning of a prompt cheaper. Context trimming leaves out earlier exchanges that the latest message does not need. They interact: a gap in the history breaks the cache for everything behind it, so trimming has to be computed per request, and sometimes it costs more than it avoids.

Why long conversations get expensive

A model keeps nothing between two requests. The application sends the system prompt, the tool definitions and every earlier message again with each turn, and all of it is billed as input tokens. If each exchange adds about the same amount of text, turn n carries about n times that amount, and the input of the whole conversation grows with the square of its length.

An invented example: each exchange adds 1,000 tokens. Turn 1 sends 1,000 input tokens and turn 10 sends 10,000. The ten turns together sent 55,000 input tokens for a conversation that holds 10,000. How the classes of tokens are priced is explained in LLM token costs.

How prompt caching works

The provider stores the processed form of the beginning of a prompt. A later request that starts with exactly the same tokens is billed for that part at a cache-read price below the input price. Four rules hold across providers, with details in each vendor's documentation:

  • The cache is a prefix. The match runs from the first token. A timestamp in the system prompt or tools in a different order, and nothing after that point is read from the cache.
  • There is a minimum length. Short prompts are not cached.
  • An entry expires. A prefix that is not read again within the provider's time window is dropped.
  • Caching is automatic or requested. Some providers cache every prefix that is long enough, without a surcharge, as OpenAI's prompt caching guide describes. Others cache where the caller sets a marker and bill the write at more than the input price, as in Anthropic's prompt caching documentation.

Where a write costs extra, a cache has its own break-even:

saving = reads × prefix tokens × (input price − cache-read price)
       − writes × prefix tokens × (cache-write price − input price)

A prefix that is written and never read again costs more than no caching. Count how many reads follow each write before marking anything.

Order the prompt from stable to volatile

Because the cache is a prefix, the order of a prompt decides what can be cached. Put tool definitions and the system prompt first, long reference documents next, the conversation after that and the latest message last. Keep values that change per request, such as the date, the user's name or retrieved passages, out of the beginning. In a chat, the history is itself a growing prefix: each turn reads the earlier turns from the cache and adds only the new end.

Why trimming works against the cache

Trimming sends the request without the earlier exchanges that have nothing to do with the latest message. Tokens left out are not billed. But taking an exchange out of the middle changes the prompt from that point on. Everything behind the gap no longer matches the cached prefix and is billed at the full input price again.

SituationWhat trimming avoids on one request
History not cachedremoved tokens × input price
History served from the cacheremoved tokens × cache-read price − tokens kept after the first gap × (input price − cache-read price)

If a model chooses what to leave out, the cost of that call is subtracted in both rows.

A worked example with invented prices

Assume an input price of 10.00 per million tokens and a cache-read price of 1.00 per million. Both are invented. A conversation holds 60,000 tokens of history, of which 30,000 are unrelated to the latest message.

CaseCalculationResult per request
No caching30,000 × 10.00 ÷ 1,000,0000.30 avoided
Cached, 8,000 tokens kept after the gap30,000 × 1.00 ÷ 1,000,000 − 8,000 × 9.00 ÷ 1,000,000 = 0.03 − 0.0720.042 more expensive
Cached, 2,000 tokens kept after the gap30,000 × 1.00 ÷ 1,000,000 − 2,000 × 9.00 ÷ 1,000,000 = 0.03 − 0.0180.012 avoided

The same conversation and the same removal give three answers. Without caching, trimming almost always pays. With caching it pays only when what is removed is large compared with what follows the gap.

What goes wrong

  • Trimming the stored history. If the application shortens the conversation it keeps, a later question about the removed part cannot be answered. Shorten only the copy that goes to the model.
  • Splitting tool calls from their results. Providers reject a tool result without its call. A user turn and everything up to the next user turn stay together or go together.
  • Judging from a message that says nothing. "Do it" or "and the second one?" gives no basis for relevance. Send the full history.
  • Removing the first exchange or the system instructions. The task is usually stated there.
  • Counting the saving at the input price. Where the history was cached, the removed tokens only cost the cache-read price.
  • Measuring cost and nothing else. A shorter context can change the answer. The calculation says what a request costs, not whether the answer is as good. Test on your own tasks before switching trimming on.

How Cureman does it

The context optimiser of Cureman runs inside its gateway, which is described in what an LLM gateway is. On conversations above a length the organisation sets, it splits the history into exchanges, always sends system messages, the first exchange and the most recent ones, and judges the rest for relevance to the latest message: by deterministic word overlap, or with a small model where that is configured. The caller's conversation is never changed. Only the copy sent upstream is shorter, and the next turn is judged afresh from the full history.

A request is trimmed only when the calculation above is positive at the price of the model that serves it. The small model is called only when the most it could avoid is several times its own cost, and that cost is deducted. If the latest message is too short to judge from, or the selector gives no usable answer, the full history goes out. If the provider rejects the shortened request, the gateway by default sends the original. One header on a request keeps its full history.

In observe mode nothing is changed: the gateway records what trimming would have avoided, so the effect is known before it is switched on. It measures cost, not answer quality. The gateway also lists large prompt prefixes that are sent again and again and never read from the cache, with the calculation and its assumption shown. The same policy can serve simple requests from a cheaper model, see LLM model routing. The module is described on the page about the context optimiser.

Questions and answers

What is prompt caching?

The model provider stores the processed beginning of a prompt. A later request that starts with exactly the same tokens is billed for that part at a cache-read price, which is lower than the normal input price. The match runs from the first token, so one changed character early in the prompt means nothing after it is read from the cache.

Does context trimming always save money?

No. If the history was not cached, every token left out saves the full input price. If the provider was serving the history from its cache, the removed tokens only cost the cache-read price, and everything after the gap loses its own cache discount. When more discount is lost than cost is avoided, the trimmed request is the more expensive one.

How do I compute whether trimming a conversation pays?

Without caching, multiply the removed tokens by the input price. With caching, multiply the removed tokens by the cache-read price and subtract the tokens kept after the first gap times the difference between input price and cache-read price. Subtract what the selection itself costs. Trim only when the result is positive.

Does trimming change the conversation the user sees?

It should not. Only the copy of the request that goes to the model is shortened. The application keeps the full history, and the next turn is judged again from that full history, so an exchange left out now is sent again as soon as a later message refers to it.

How does Cureman decide whether to trim a request?

Its context optimiser computes the avoided input cost for each request at the price of the model that serves it, with the cache case priced at the cache-read rate minus the discount lost behind the gap, and deducts the cost of a selector call. It shortens the upstream copy only when the result is positive, and in observe mode it only records what trimming would have avoided.