AI costs

Reducing AI costs: the levers, from model choice to context

Updated 6 min readMarkdown
In short

A model bill is tokens times price, so it falls when fewer tokens are sent, when they are sent to a cheaper model, or when they are billed at a lower rate. The levers are the choice of model per request, the length of answers, prompt caching, the conversation history sent with each turn, repeated and failed requests, the provider's cheaper processing tiers, and budgets that stop a runaway job. Start by pricing the usage reports, because the largest line decides which lever comes first. Every lever except budgets can change the answers, so each is measured on your own traffic before it is switched on.

You reduce AI costs by sending fewer tokens, sending them to a cheaper model, or having them billed at a lower rate. Everything else is a variant of these three. Which one matters for you depends on where the money goes today, so the first step is a breakdown of the bill and not a change to the code.

Where the cost comes from

Providers bill four kinds of tokens per model: uncached input, output, input read from the prompt cache and input written to it. Output usually costs a multiple of input, and cache reads a fraction of it. The details are in LLM token costs explained. Three facts follow:

  • A long prompt sent many times costs more than a long answer given once.
  • In a conversation, every turn sends the whole history again as input.
  • The price per token differs widely between the largest and the smallest model of the same provider.

The levers at a glance

LeverWhat it changesWhat to know before pulling it
Model choice and routingPrice per tokenWhich requests are simple enough
Output lengthNumber of output tokensWhether answers are longer than anyone reads
Prompt cachingRate for repeated inputWhich prefixes repeat, and how often within the cache lifetime
ContextNumber of input tokens per turnHow long conversations get, and whether they are cached
Repeats and retriesNumber of requestsHow many requests are identical or failed
Processing tierRate for work that can waitWhich jobs need no immediate answer
Seat or tokensThe billing model per personHow much each person uses
Budgets and limitsThe worst caseWho may spend how much

Step 1: measure before changing anything

Read each provider's usage report, price every token type and break the total down by model, project and day. The method is in how to audit AI spend. The report tells you which lever is worth the effort. It cannot tell you which requests were simple or which prompts repeat. That needs metering per request, which is what an LLM gateway does.

Step 2: choose the model per request

Most applications send every request to one model, chosen for the hardest case. Classify requests by what they need, for example extraction, classification, a short answer or a long piece of reasoning, and serve the simple ones from a smaller model of the same provider.

A worked example with invented, round figures. A month has 200 million input tokens and 20 million output tokens on a large model at invented prices of 10.00 and 40.00 per million.

CalculationCost
Everything on the large model200 × 10.00 + 20 × 40.002,800.00
Half of the tokens on a small model at 1.00 and 4.00100 × 10.00 + 10 × 40.00 + 100 × 1.00 + 10 × 4.001,540.00
Difference1,260.00

The half is an assumption of the example. Which share of your requests can move is exactly what has to be measured, and the cost of classifying requests is deducted from the difference. See LLM model routing.

Step 3: shorten what comes back

Output is the dearest token type. Set an upper limit on output tokens per request type, ask for the format you need (a value, a list, a JSON object) instead of an explanation, and check whether reasoning or thinking modes are switched on where nobody needs them.

Step 4: cache what repeats

System instructions, tool definitions and reference documents are often identical from request to request. With prompt caching the provider bills such a prefix at a lower rate when it is read again within the cache lifetime. It works only when the prefix is byte for byte the same, so anything that changes, such as a date, a user name or a random order of tools, belongs after the stable part.

Step 5: send less history

A long conversation pays for its whole history on every turn. Shortening the copy sent to the model avoids those tokens. It only pays when the calculation is positive, and with caching it often is not, because removing a part in the middle makes everything after it full-price input again. The calculation is in prompt caching and context.

Step 6: remove repeats and use the slower tier

Look for requests that are sent twice: a retry loop without a limit, a job that runs again over unchanged data, a page that asks the same question on every load. Work that needs no immediate answer, such as nightly classification or evaluation runs, can go through a provider's batch processing where one is offered, which is billed below the standard rate.

Step 7: check the billing model per person

For employees who chat with a model, compare the seat licence with what their tokens would cost. A light user can be cheaper by the token, a heavy user by the seat. The comparison is in seat licence or pay per token.

Step 8: set budgets

A budget per key, per user and per month does not lower the price of anything. It bounds the damage of a loop or a leaked key. See AI budgets and limits.

What goes wrong

  • Optimising the small line. Effort goes into prompts of a feature that causes a sliver of the cost.
  • Reporting a ceiling as a saving. A repricing of all tokens on the cheapest model is an upper bound.
  • Measuring cost only. A cheaper request that produces a worse answer costs somewhere else. Test on your own tasks.

How Cureman does it

Cureman covers these levers in modules that can be booked separately. The AI spend audit reads the providers' usage reports through read-only connectors, prices every token type against a model price list in which unverified prices are labelled, and shows what the same tokens would cost on cheaper models as a ceiling and not as a saving.

The AI gateway sits between applications and the provider, meters every request and classifies it. A policy maps simple requests to a cheaper model of the same provider. In observe mode it only records what would have happened; it changes requests after a person has switched the policy to enforce. The saving is computed per request as the cost on the requested model minus the cost on the served model, minus the classifier calls. Keys carry model lists and monthly budgets per key and per user.

The context optimiser shortens only the copy of a conversation sent upstream, never the caller's history, and only when that is computed to pay, with the cache-read price taken into account. Prompt insights show the task mix and the spend per person, and the AI workspace lets employees chat by the token on the organisation's own provider key. None of these measures answer quality. That check stays with you.

Questions and answers

What is the fastest way to lower a model bill?

Find the largest line first. Price the provider's usage report by model, project and token type, and look at the top three rows. If one large model carries most of the cost, model choice is the lever. If input dominates and little of it is read from cache, caching and context are. Work on anything else comes later.

Does a cheaper model always save money?

Per token yes, per task not always. A smaller model can need more attempts, longer prompts or a human correction, and none of that is on the provider's invoice. Move the requests that are simple, record what the same tokens would have cost on the model that was asked for, and check the answers on your own tasks before you rely on the difference.

How much can be saved on AI costs?

Nobody can say without your usage data, and a figure offered without it is a guess. What can be computed is a ceiling, namely what the same tokens would cost on a cheaper model or with every repeated prefix cached. What is actually saved is known only after requests have been metered one by one.

Does shortening the context always pay?

No. Without prompt caching, tokens left out are avoided at the full input price. With caching, the history would have been billed at the lower cache-read price, and removing a part in the middle means that everything after the gap is billed at the full price again. Whether it pays is a calculation per request.

Do budgets reduce AI costs?

They limit them. A budget per key or per user stops a loop, a leaked key or a forgotten job before the month's invoice shows it. It does not make a single request cheaper. Budgets belong in place from the first day, the other levers after measuring.