Context optimiser: long conversations sent with less history
The context optimiser of Cureman works inside the AI gateway. On a long conversation it selects the earlier exchanges that matter for the latest message and leaves the rest out of the request that goes to the provider. The caller's own history is never changed, so the next turn starts from the full conversation again. It shortens only when the avoided input cost exceeds what prompt caching would have saved and what the selection costs, and it starts in observe mode, where it measures and changes nothing.
A chat request carries the whole conversation every time, and every token of it is billed as input again on each turn. On a long conversation much of that history has nothing to do with the latest message. The context optimiser decides which earlier exchanges the current request needs and sends the provider a copy of the request without the rest.
What it works on
The optimiser looks at requests that pass through the AI gateway, in the Anthropic Messages, OpenAI Chat Completions and OpenAI Responses formats. It splits a conversation into exchanges: one user turn with everything up to the next user turn, so that a tool call and its result stay together. Only conversations whose history exceeds a minimum size set in the policy are considered. A request that relies on a conversation kept on the provider's side is left alone.
What is always sent
- System and developer messages.
- The first exchange of the conversation.
- The most recent exchanges; the policy sets how many.
- Every earlier exchange the selector judges relevant to the latest message.
An exchange is sent whole or left out whole. It is never split.
How the selection is made
There are two selectors. The deterministic one compares the distinctive words of the user's recent turns with each earlier exchange and keeps those that share enough of them. The other is a small model on the same upstream: it is given the latest message and a shortened, numbered list of earlier exchanges and returns the numbers to keep. The conversation is handed to it as data, with the instruction to treat it as data, and its answer has to match a fixed shape.
When a selector cannot judge, the full history is sent. That is the case when the latest message is too short to compare, or when the small model returns no usable answer.
When it shortens: the calculation
| Situation | Input cost avoided |
|---|---|
| The history would have been billed at the full input price | Tokens left out times the input price |
| The provider would have served the history from its prompt cache | Tokens left out times the cache-read price, minus the cache discount lost on everything that follows the first gap |
A request is shortened only when this figure is positive. Leaving something out of the middle of a conversation ends the cached prefix at that point, so on a well-cached conversation trimming often does not pay, and then nothing is left out. The planning works on estimates; once the provider reports the tokens it billed, the stored figures are scaled to that count. The article Prompt caching and context explains the trade-off.
What never changes
The caller's history is not touched. Only the copy that goes upstream is shorter, and the next turn is judged from the full conversation again. If the provider rejects a shortened request, the gateway by default sends the request in full. A caller can keep the full history for a single request with the header x-gateway-context: off, and a policy can exempt tags. A response header says when a request was shortened and how many messages were left out.
Observe first
The optimiser is set per routing policy and has three modes: off, observe and enforce. In observe every request is sent with its full history, and the gateway records which exchanges it would have left out and what that would have saved, using the deterministic selector. The result is a finding that shows the requests, the tokens and the amount, with its calculation.
That finding has an action plan with one step: switching the optimiser to enforce. The step is prepared and takes effect only after a person approves it in the list of decisions. In enforce the model answers without having seen what was left out, and the optimiser measures cost, not answer quality. Read the observed figures and decide for which workloads that is acceptable.
What it needs and what it measures
The context optimiser requires the AI gateway. It meters the input cost of the history left out on shortened requests, minus the cost of selector calls.
Related reading
Questions and answers
Does the context optimiser change my application's conversation?
- No. Only the copy of the request that is sent to the provider is shorter. The response goes back untouched, the application appends it to its own full history, and the next request is judged afresh from that full history. An exchange that was left out is sent again as soon as a later message refers to it.
Can leaving out history change the model's answer?
- Yes, it can. In enforce mode the model answers without having seen the exchanges that were left out. The selector keeps an exchange when it is unsure, and the full history is sent when it cannot judge at all. The module measures cost and does not measure answer quality, which is why it starts in observe mode and why single requests and tags can be exempted.
How does it work together with prompt caching?
- The calculation takes caching into account. Where the provider would have served the history from its prompt cache, the tokens left out are valued at the cache-read price only, and the discount lost on everything after the first gap is subtracted. If the result is not positive, the request is sent in full.
What does the selection itself cost?
- The deterministic selector compares words and costs nothing. The small-model selector makes one call on the same upstream. That call is made only when leaving out every candidate would save several times what it costs, it is metered like any other request, and its cost is deducted from the saving.