AI costs

AI budgets and limits: per key, per user, per model

Updated 6 min readMarkdown
In short

Token billing has no ceiling of its own. A budget adds one by comparing the cost recorded so far in the month with a limit before each request is forwarded, and refusing the request once the limit is reached. Three limits do most of the work. A monthly budget per key contains one application, a monthly budget per user contains one person on a shared key, and a model allow-list keeps the most expensive model from becoming the default. A budget is enforced between requests, so the request that crosses the line is still served, and the application has to treat the refusal as final and not as a rate limit to retry.

Token billing has no ceiling of its own: an agent caught in a loop or a job that retries without end spends until somebody reads the invoice. A budget is a ceiling that is checked before each request. The cost recorded so far in the period is compared with a limit, and a request over the limit is refused. Three limits do most of the work: a monthly budget per key, a monthly budget per user, and a list of the models a key may ask for.

What each limit stops

LimitApplies toWhat it stopsWhat it does not stop
Monthly budget per keyOne application or serviceA runaway job or loop in that applicationOne person using up what was meant for everyone
Monthly budget per userOne person on a keyA single heavy user of a shared applicationRequests that name no user
Model allow-listOne keyThe most expensive model becoming the defaultHigh volume on an allowed model
Maximum output tokensOne requestOne very long answerMany requests
Spend limit at the providerThe whole account or projectEverything, as the last line of defenceNothing selectively: every application stops at once

The limits build on attribution. A budget per user needs a user on each request, as described in allocating AI costs to teams.

How a budget check works

  1. Every completed request is recorded with its cost: its token counts times the price of the model that answered.
  2. Before a new request is forwarded, the recorded cost of the key, or of the key and the user, is summed from the start of the period.
  3. If the sum has reached the limit, the request is refused. Otherwise it goes out.

Three things follow from this order:

  • The check sits between requests. What a request costs is known only when the answer is complete, and with streaming that is at the very end. The request that crosses the limit is served in full.
  • The overshoot is what is in flight. With parallel traffic, every request that started before the sum reached the limit completes. A system that remembers the sum briefly lets a few more seconds of spend through.
  • The period needs a definition. A calendar month in which time zone? A budget that resets at midnight UTC resets at one or two in the morning in Berlin.

What happens at the limit

A refused request usually comes back with HTTP status 429 and a message. That status is also what providers use for rate limits, and most SDKs retry it automatically with a growing delay. A rate limit clears in seconds. A budget does not. Client code should therefore:

  1. Read the message and tell a budget refusal from a rate limit.
  2. Not retry a budget refusal.
  3. In an interactive application, tell the person in one sentence what happened, when the budget resets and whom to ask.
  4. In a background job, stop and alert its owner.

At the limit a system can refuse, only warn, or fall back to a cheaper model. Only refusing bounds spend. Decide per key whether it may ever be stopped: a feature your customers use wants a generous budget and a person watching it, not a tight hard stop.

How to set the amounts

  1. Measure first. Run a month without a limit and record cost per key and per user.
  2. Key budget. Take a typical month and add the growth you would accept without a conversation. The budget is there to catch what is abnormal, not to ration normal work.
  3. Budget per user. Where a seat licence is the alternative, its monthly price is a natural ceiling: somebody who would cost more per token is better served by a seat. The calculation is in AI chat: seat licence or pay per token.
  4. Check the two against each other. Multiply the users by the budget per user and compare with the key budget.

A worked example with invented amounts. A shared chat key has 40 users, a budget of 20.00 per user and a key budget of 500.00. 40 × 20.00 = 800.00, which is more than 500.00. If 25 people use their full 20.00, the key is at its limit and the other 15 are refused although they have spent nothing. Either raise the key budget to 800.00 or accept that it is the real limit and say so. Without the budget per user, one person with a script could spend the 500.00 alone.

The model allow-list

Between a provider's smallest and largest model the price per token differs by a multiple. An allow-list decides which models a key may ask for at all. A request for any other model is refused with a message that names the allowed ones, and a chat front end shows only those in its model picker. Routing is a different tool: it serves a simple request from a cheaper model than the one asked for. See LLM model routing.

What goes wrong

  • One budget for everything. A nightly job fills it, and the assistant your customers use stops.
  • A budget per user without a user. Requests that carry no identifier count against nobody.
  • Retrying the refusal. A retry loop on status 429 runs until the first of the month.
  • A different currency. A budget in euros against a bill in dollars moves with the exchange rate.
  • Models without a price. A request that is recorded at zero never reaches a limit.

How Cureman does it

In the gateway of Cureman, every key can carry three optional settings: a list of allowed models, a monthly budget for the key and a monthly budget for each end user of the key. Budgets are in US dollars per calendar month, in UTC. Before a request is forwarded, the gateway sums the key's metered cost for the month, including the classifier and selector calls made for its requests, and compares it with the budget. The budget per user applies to requests that name a user, through a header, the end-user field of the request body or the signed-in user a chat front end passes on.

At the limit the request is answered with status 429 and a message that says whether the key's or the person's budget is used up, and that it resets on the first of the month or can be raised by an administrator. The sum is remembered for some seconds and a request under way is completed, so a limit can be passed by what is spent in that moment. A request for a model that is not on the key's list is refused with the list of enabled models, and the model listing that chat front ends call returns only those. A model without a price in the price list is metered at zero and does not count towards a budget until its price is entered.

The same settings apply to the key of the employee workspace, where the budget per person and the model list are set. What a gateway does beyond this is explained in what an LLM gateway is, and the module on the page about the AI gateway.

Questions and answers

What happens when an API key reaches its AI budget?

The next request is refused before it reaches the model provider, usually with HTTP status 429 and a message. Nothing is billed for a refused request. The key works again when the period resets or when someone raises the limit, so the application should show a clear message and stop instead of retrying.

Can an AI budget be exceeded?

Yes, by a little. The cost of a request is known only when its answer is complete, so the check happens before a request and not during it. The request that crosses the limit is served in full, and so are requests that were already under way in parallel. The overshoot is bounded by what is in flight at that moment.

What is the difference between a budget and a rate limit?

A rate limit caps requests or tokens per minute, protects capacity and clears within seconds. A budget caps money per month and stays reached until the month ends or the limit is raised. Both are often answered with status 429, which is why client code has to read the message and must not retry a budget refusal.

How should a monthly budget per user be set?

Measure a month without a limit first, then set the amount above what normal work costs, so that it catches the unusual case and does not ration ordinary use. Where a seat licence is the alternative, its monthly price is a natural ceiling, because somebody who costs more per token than a seat is better served by the seat.

How do budgets work in Cureman?

Each gateway key can have a list of allowed models, a monthly budget for the key and a monthly budget per end user. Before forwarding a request the gateway compares the key's metered cost for the calendar month with the budget and answers with status 429 once it is reached. A request for a model that is not on the list is refused and told which models are enabled.