What is an LLM gateway, and when do you need one?
An LLM gateway is a proxy that sits between an organisation's applications and its model providers. Applications keep their SDK and change two settings, the base URL and the key. The gateway holds the provider keys, forwards each request to an upstream, and records per request what a provider's usage report only shows as a total, namely the model, the token counts, the cost, the calling feature and the person. On that record it can enforce budgets and model lists and, if you choose, route requests. You need one when several applications or teams call models and the provider's report no longer answers who spent what on which kind of request.
An LLM gateway is a proxy between your applications and the model providers. Applications send their requests to it instead of to the provider, it forwards them with a provider key that only it holds, and it records every request on the way. You need one when several applications or teams call models and the provider's usage report no longer answers who spent what, and on what.
How an LLM gateway works
A request passes through the gateway in six steps:
- The application sends its request to the gateway's address, in the provider's own API format, with a gateway key.
- The gateway identifies the key and looks up what belongs to it: the upstream, the allowed models, the budget, the policy.
- It checks the request against those rules and refuses what is not allowed, with an error in the format the SDK expects.
- It forwards the request to the upstream with the provider key.
- It passes the answer back, streamed answers included, and reads the token usage as it goes by.
- It writes one record for the request.
For the application the change is two settings:
before base URL https://api.provider.example/v1 key provider key
after base URL https://gateway.example/v1 key gateway keyWhat an LLM gateway meters
A provider's usage report gives totals per day, model and project. A gateway sees each request, so it can record what the report leaves out.
| Recorded per request | What it is for |
|---|---|
| Requested model and served model | The basis of every price comparison |
| Input, output, cache-read and cache-write tokens | Cost, computed from the price list |
| Key, tag and end user | Attribution to an application, a feature, a person |
| Status and latency | Errors, retries, slow upstreams |
| Features of the request: size, number of tools, length of the conversation | Classifying how demanding the request was |
| A hash of the prompt's static start and of the whole request | Finding prefixes that repeat without being cached, and exact repeats |
None of this needs the text of a prompt or an answer. Whether a gateway stores content anyway is a product decision that you should ask about. How the token counts turn into money is covered in LLM token costs explained.
Upstreams and keys
An upstream is where the gateway sends requests: a provider's API or any endpoint that speaks the same format, such as a cloud deployment of a model or a model you host. The upstream holds the provider key.
A gateway key is what an application presents. It belongs to one upstream and carries the rules for its holder. This split is the main operational gain:
- The provider key lives in one place. Rotating it touches the gateway and no application.
- Every application or service gets its own key, which can be revoked alone.
- A key can be limited to certain models and given a monthly budget, for the key as a whole and for each end user. What should happen at the limit is discussed in AI budgets and limits.
- Spend falls into place by key before anyone has tagged anything. Finer attribution by feature and person is the subject of allocating AI costs to teams.
API formats
Applications talk to models in a small number of formats, among them Anthropic Messages, OpenAI Chat Completions and OpenAI Responses. A gateway either passes a format through to an upstream that speaks it, or translates between formats. Pass-through keeps every feature of the provider's API, because the request arrives as it was written. Translation lets one application reach any provider, at the price that features which exist in only one format get lost or approximated.
Streaming needs attention in either case. The gateway must pass the stream through as it arrives and still read the usage, which providers report in events of the stream itself. In the Chat Completions format usage is only included in a stream when the request asks for it (stream_options.include_usage), so a metering gateway has to set that option.
What a gateway can do beyond metering
Once every request passes one point, three controls become possible:
- Limits. Budgets and model lists are enforced before a request costs anything.
- Observation. Each request is classified, and the gateway records what a cheaper model would have cost for the same tokens, without changing the answer.
- Routing. Under a policy you set, requests classified as simple are served by a cheaper model. How that works and how to measure it is explained in LLM model routing.
The order matters. Meter first, observe for a while, and decide about routing on the recorded figures.
When you need one
| Situation | What the gateway adds |
|---|---|
| Provider keys are spread over applications and people | One place for provider keys, one revocable key per caller |
| The usage report shows one total per project | Cost per feature, team and person |
| A runaway job or a popular feature can exhaust the month | Budgets that refuse requests at the limit |
| Someone proposes a cheaper model for part of the traffic | Measured figures per request before anything changes |
| More than one provider or deployment is in use | The same record for all of them |
With one application, one provider and one key, the usage report is enough. Start with an audit of AI spend and add a gateway when its questions come up.
What it costs you
A gateway is infrastructure in the path of every request.
- A further hop. Each request takes one more network step, and classification that calls a model adds a round trip.
- Availability. If the gateway is down, every application behind it is down.
- Trust. Prompts pass through it and provider keys are stored in it.
- Metering is not the invoice. Compare a day of metered cost with the provider's own report before relying on it.
Ask of any gateway what it stores, what it changes in a request, and what it does when the upstream rejects a request it changed.
How Cureman does it
The gateway of Cureman offers Anthropic Messages, OpenAI Chat Completions and OpenAI Responses endpoints, streaming included, and passes each format through to an upstream that speaks it: Anthropic, OpenAI, Azure OpenAI or any OpenAI-compatible address on public HTTPS. The organisation's provider key is stored encrypted; of a gateway key only a hash is kept. A key can carry a model list and monthly budgets per key and per end user, and a request over budget is refused with a message that says so. Every request is recorded with requested and served model, the four token counts, cost from the model price list, status, latency, tag and user, which callers can send as x-gateway-tag and x-gateway-user. A model that is not in the price list is recorded with its tokens, without a cost, and is never routed. A policy starts in observe mode, where requests pass through as written. Routing happens only in enforce mode, a request can opt out with a header, and unless the policy says otherwise a changed request that the provider rejects is sent again exactly as asked. The module is described on the page about the AI gateway.
Questions and answers
What is an LLM gateway?
- It is a proxy between applications and model providers. Applications send their requests to the gateway in the provider's own API format, the gateway forwards them with the provider key it holds and passes the answer back. On the way it records tokens, cost and who called, and it can apply budgets, model lists and routing rules.
Do applications have to be rewritten to use a gateway?
- No, as long as the gateway speaks the API format the application already uses. The application keeps its SDK and changes two settings, the base URL and the API key. Labels for cost attribution, such as the calling feature or the end user, are optional headers.
Does an LLM gateway store prompts and answers?
- That depends on the product, and it is the first question to ask. Metering needs token counts, not content. Cureman stores token counts, features of the request, hashes, the task type and the user reference, and stores no prompts or responses unless the organisation switches prompt samples on.
Does a gateway change requests or answers?
- A gateway that only meters passes both through unchanged. One that routes sends a request to a different model than the application named, which is a change the organisation has to switch on knowingly. A sound design has a mode that only observes, a way for a single request to opt out, and a retry of the original request if the changed one is rejected.
When is an LLM gateway not needed?
- When one application calls one provider with one key, the provider's usage report already answers what is spent. A gateway earns its place when there are several applications, teams or providers, when cost has to be attributed to features or people, when budgets have to hold, or when you want to measure what routing would do before deciding on it.