AI gateway: one endpoint that meters, classifies and routes
The AI gateway of Cureman is one endpoint in front of every model provider. Applications keep their SDK and change the base URL and the key. The gateway speaks Anthropic Messages, OpenAI Chat Completions and OpenAI Responses, forwards to any compatible upstream with the organisation's own keys, records tokens and cost for every request and classifies it. Under a routing policy the organisation controls, simple requests are served from a cheaper model. It never changes a caller's conversation, and it starts in observe mode, where it measures and changes nothing.
Applications keep the SDK they already use and change two settings: the base URL and the API key. From then on every model request passes through one endpoint that records what it cost, labels what kind of request it was and, where the organisation's policy says so, serves a simple request from a cheaper model.
What it takes in
| Format | Endpoint under the gateway's base URL |
|---|---|
| Anthropic Messages | /v1/messages |
| OpenAI Chat Completions | /v1/chat/completions |
| OpenAI Responses | /v1/responses |
| Model list for SDKs and chat front ends | /v1/models |
Streaming works on all three. The gateway forwards to an upstream: a provider, or any endpoint compatible with one of these formats, called with the organisation's own API key. That key is stored encrypted and not shown again. An application authenticates with a gateway key, which is shown once and stored as a hash. A gateway key belongs to one upstream and can carry a model allow-list, a monthly budget, a budget per end user and a routing policy. An organisation can also put the endpoint on a host name of its own.
What it records
For every request the gateway stores who sent it (key, tag, user reference), the model asked for and the model that served it, input, output, cache-read and cache-write tokens, the cost computed from the model price list, the complexity class, the task type, latency and status. It stores no prompt or response text unless the organisation switches prompt samples on.
How routing decides
- The gateway reads features off the request without calling a model: input size, number of tools, code in the request, wording that points to reasoning or to a simple task, whether extended thinking was asked for, the length of the conversation, attachments and the output limit.
- A scorer turns the features into a score and lists its reasons. The policy's two thresholds sort the score into simple, standard or complex.
- Where the policy asks for it, a small model on the same upstream labels the request instead, and only when the request is expensive enough for that call to pay. Its cost is recorded on the request and deducted from the saving.
- The policy names a target model per class: an alternative listed for the requested model first, then the provider's default. The target is used only when it is in the price list, offered by the upstream and cheaper than what was asked for. The gateway never routes to a dearer model.
- If the provider rejects the changed request, the gateway by default sends exactly what was asked for.
A policy runs in one of three modes. Off forwards and meters without classifying. In observe every request is served by the model the application asked for, and the gateway records what the policy would have picked and what the same tokens would have cost there. In enforce it serves the request from that model. A response header names the model that served the request. The method is described in LLM model routing.
What it finds
| Finding | How it is computed |
|---|---|
| Routing would pay | Requests the policy would have served from a cheaper model: their cost on the requested models minus the same tokens on the target models |
| Repeated prompt prefixes | Large identical system prompts and tool definitions sent at the full input price, repriced at the cache-read price, with the assumed hit rate stated in the calculation |
| Identical requests | Requests that repeat an earlier request byte for byte, and what they cost |
| Background jobs | Workloads whose tag marks them as running without a user waiting, repriced at the provider's batch price |
Each finding shows its calculation and the period of traffic it rests on.
Budgets and limits
A request for a model outside a key's list of allowed models is refused. Once the metered cost of the month has reached the key's budget or an end user's budget, further requests are refused until the next month begins or an administrator raises it. See AI budgets and limits.
From observation to enforcement
A policy starts in observe. The routing finding has an action plan whose step switches policies from observe to enforce. The step is prepared, shown and takes effect only after a person approves it in the list of decisions. Nothing in the product switches a routing policy by itself.
What it needs and what it measures
The gateway requires no other module. It needs an upstream with the organisation's own key and prices for the models in use. The context optimiser, prompt insights and the AI workspace build on it. It meters the price difference between the requested and the served model on routed requests, minus the cost of classifier calls.
Related reading
Questions and answers
Do applications have to be rewritten to use the gateway?
- No. An application keeps its Anthropic or OpenAI SDK and changes two settings, the base URL and the API key. Two optional request headers add a tag for the calling feature and the end user the request is for, which is what cost attribution and per-user budgets use.
Does the gateway change requests or responses?
- In observe mode a request is forwarded as it came, apart from asking the provider to report token usage on a stream. In enforce mode the gateway replaces the model name on requests the policy sends to a cheaper model. It never changes a caller's conversation, and responses stream back byte for byte. A caller can exempt a single request with a header, and a policy can exempt tags.
Does the gateway store prompts and responses?
- No. It stores token counts, features of the request, hashes, the task type and the user reference. Prompt and response text is not stored unless the organisation switches prompt samples on, in which case the beginning of each request's system prompt is kept.
Which providers can it forward to?
- Any endpoint that speaks the Anthropic Messages format or the OpenAI formats over public HTTPS, called with the organisation's own API key. That covers the providers themselves and compatible endpoints of other vendors. A model has to be in the model price list to be costed and routed.
What happens when a budget is used up?
- The gateway refuses further requests for that key, or for that end user, with an error that says so. The budget counts the metered cost of the calendar month and starts again on the first of the next month, or an administrator raises it.