LLM model routing: simple requests on a cheaper model
Model routing serves a request from a cheaper model than the one the application asked for when the request is simple enough. A router classifies each request, either by rules over features it can read off the request or by asking a small model, and a policy maps each class to a target model. The price difference is arithmetic. Take the tokens of each routed request, price them on the requested model and on the model that served, subtract, and deduct what the classification cost. That figure measures cost and says nothing about answer quality, so routing is first observed without changing anything and then switched on for one workload at a time.
Model routing serves a request from a cheaper model than the one the application asked for, when the request is simple enough for it. It rests on three parts: a classification of each request, a policy that maps classes to models, and a measurement that prices the same tokens on both models.
What model routing changes
An application names a model in every request. Most name one model for everything, chosen for the hardest case, so picking one of five categories is answered by the same model as restructuring a codebase. A router sits in the path of the request, for example in an LLM gateway, and replaces the model name where its policy says so. Everything else in the request stays as written.
Routing for cost only goes down: to a model that is cheaper than the requested one. Because input and output are priced separately, "cheaper" needs a definition, for instance a blended price that weights input and output in a fixed ratio. The background is in LLM token costs explained.
Classifying a request
A request can be judged in two ways, which can be combined.
Rules over features. A scorer reads features off the request without calling any model and adds them up to a score.
| Feature of the request | Effect on the score |
|---|---|
| Large input | Raises it |
| Tools offered to the model | Raises it, per tool |
| Code in the latest message | Raises it |
| Wording such as "root cause", "trade-offs", "step by step" | Raises it |
| Wording such as "classify", "extract", "translate", "yes or no" | Lowers it |
| Extended thinking requested by the caller | Raises it |
| Long conversation, attached images or documents | Raises it |
| A large output limit | Raises it |
| A very small output limit, a small structured-output task | Lowers it |
Rules add no latency and no cost, and every decision can be shown with its reasons. Their weakness is wording they have no rule for, and prompts in a language the rules were not written in.
A small model as classifier. The latest message, an excerpt of the system prompt and a few facts about the request go to a small, fast model, which returns one of the classes. It reads meaning where rules read keywords. It costs a few hundred tokens and one round trip, so it should run only where it can pay: when a cheaper target exists and the request is expensive enough that classifying it is a small part of the cost. The classifier must treat the request as data and not follow instructions in it. When unsure, it should choose the higher class. And if it is slow or answers with something unexpected, the rule-based result stands.
Thresholds and the policy
Two thresholds cut the score into three classes: simple below the lower one, complex above the upper one, standard in between. The policy then says where each class goes.
- Name a target model for the classes that may move, per provider, and where needed per requested model.
- Leave the complex class on the requested model.
- Allow a target only when it is in the price list, comes from the same provider, is offered by the upstream and is cheaper than what was requested.
- List the tags that are never routed, and let a caller opt a single request out.
- Keep a fallback: if the upstream rejects the routed request, send the original one.
A lower threshold for "simple" moves fewer requests and takes less risk; a higher one moves more.
Observe before you enforce
A router should have a mode in which it classifies every request, records which model the policy would have chosen and what the same tokens would have cost there, and still serves the request from the model the application asked for. The application notices nothing, and after a week or two you have figures from your traffic: how many requests fall into each class, on which workloads, and what the price difference would have been. That is the basis for the decision.
Computing the price difference
For each routed request:
difference = cost of its tokens on the requested model
− cost of its tokens on the model that served it
net = sum of the differences − cost of all classifier callsA worked example with invented prices. The requested model costs 4.00 per million input tokens and 20.00 per million output tokens, the target 0.50 and 2.50. In one week, 10,000 requests are classified as simple, with 30 million input tokens and 4 million output tokens together. A small model classified 2,000 of them, at about 900 input and 30 output tokens per call.
| Item | Calculation | Amount |
|---|---|---|
| Tokens on the requested model | 30 × 4.00 + 4 × 20.00 | 200.00 |
| Tokens on the target model | 30 × 0.50 + 4 × 2.50 | 25.00 |
| Difference | 200.00 − 25.00 | 175.00 |
| Classifier calls | 1.8 × 0.50 + 0.06 × 2.50 | 1.05 |
| Net for the week | 175.00 − 1.05 | 173.95 |
A smaller model may write a longer or shorter answer than the larger one would have, so the figure is exact for the cost that occurred and approximate for the cost that was avoided.
What the figure does not show
The price difference measures cost, not whether the answers were as good. That has to be checked separately:
- Switch routing on for one tag or workload at a time, starting where a weaker answer does little harm.
- Take a sample of real requests, run them on both models and have the people who own the feature compare the answers.
- Watch what the feature already measures: error rates, retries, corrections by users, failed validations of structured output.
- Exclude workloads where a mistake is costly.
What goes wrong
- Enforcing without observing. The first figures then come from production answers.
- Counting the gross difference. Classifier calls are a cost of routing and belong in the calculation.
- Routing across a prompt cache. Caches are kept per model. A conversation whose turns alternate between two models pays to fill both. See prompt caching and context.
- Assuming equal behaviour. A smaller model may lack a parameter, a tool feature or the context size. Without a fallback the request fails.
How Cureman does it
In the gateway of Cureman a rule-based scorer classifies every request from the features above and records the score with its reasons. A policy can add a small model on the same upstream as classifier. It is called only when a cheaper target exists and the request is estimated to cost a multiple of the classification, and its cost is recorded on the request and deducted. The model labels the request and deterministic code computes every amount. The policy holds the thresholds, the targets per provider and per requested model, the excluded tags and the fallback, and it routes only to a priced, cheaper model of the same provider that the upstream offers. In observe mode the recommended model and the cost of the same tokens on it are recorded and nothing is changed. The resulting finding shows its calculation and states that quality is not measured, and its action plan switches the policy to enforce only after a person approves. The module is described on the page about the AI gateway.
Questions and answers
What is LLM model routing?
- It is the practice of serving a request from a different model than the one the application named, chosen by how demanding the request is. In cost work it means sending simple requests such as classification, extraction or short rewrites to a smaller, cheaper model of the same provider and leaving demanding requests where they are.
How does a router decide that a request is simple?
- Either by rules or by a model. Rules compute a score from features of the request, such as its size, the number of tools, code in the prompt, wording that signals reasoning, and the output limit. A model classifier sends the latest message to a small model that returns a class. Rules cost nothing and can be explained line by line. A model reads wording better and costs tokens and a round trip.
How is the saving from model routing measured?
- Per request and from metered tokens. Price the tokens of each routed request on the model that was requested and on the model that served it, and subtract. Then deduct the cost of the classifier calls. Before routing is switched on, the same calculation on observed requests shows what it would have saved.
Does model routing lower answer quality?
- It can, and the cost figure does not show it. A smaller model may answer a request less well, and a classifier can mistake a demanding request for a simple one. The safeguards are to observe before enforcing, to switch routing on for one workload at a time, to compare answers on a sample of your own requests, and to exclude workloads where a mistake is costly.
Can a single request or workload be kept out of routing?
- It should be possible in any router. The usual means are a list of tags that are never routed, a header with which a caller opts a request out, and a model the policy has no cheaper target for. Requests that are excluded are still metered.