Why routing to a cheaper model sometimes costs more

The obvious way to cut an inference bill is to send easy work to cheaper models. Classification does not need a reasoning model. Summarisation does not need a reasoning model. Point them somewhere smaller and watch the per-call price fall.

Then the invoice arrives and it has gone up.

What happened

The cache is per model. A prompt cached against one model is not cached against another — the second model has never seen it. So the moment you route one call in a conversation elsewhere, you pay full fresh-input price at the new model, and, depending on how the provider ages entries, you may pay it again at the original model when the conversation returns.

For a workload with a large stable prefix — the shape the previous post describes — that penalty is large. Large enough, routinely, to exceed the difference between the two models' rates.

Why each decision still looked correct

Because it was, locally. The cheaper model genuinely is cheaper per token for that call. What the decision omitted is that the choice has a cost paid by the next call, and by the call after that, and that cost does not appear anywhere in the comparison that produced the decision.

This is an accounting failure before it is a routing failure. Judge a switch on its own per-call price and you will switch far too often.

What has to be in the decision

• The state of the cache for this prompt, at each candidate model. A switch into a cold cache is a different decision from a switch into a warm one.
• How much of the prompt is stable. The larger the fixed prefix, the more a switch costs and the higher the bar it must clear.
• How many calls follow. Paying a cache miss once to serve a thousand subsequent calls more cheaply is good. Paying it to serve three is not.
• Whether the workload is stable enough to be worth pinning. A workload that has settled is better served by one model consistently than by a router re-deciding every turn.

The consequence for how we build

It is why the Gateway does not route on a static task-to-model map, and why compression is KV-compatible rather than merely aggressive. A compressor that rewrites the stable prefix invalidates the cache on every call, and then congratulates itself on a shorter prompt while the bill climbs. Shortening the prompt and keeping the cache are the same problem, and solving only the first one makes things worse.

It is also why the Ledger prices every call twice. A routing system that cannot show the counterfactual is asking you to take exactly this class of error on faith.