You Probably Don't Need an LLM Router

Manifest published a post this month called Everyone is building LLM routers, we deprecated ours. They built a router that classified every request into a complexity tier and sent it to the cheapest capable model across four providers. Four months and thousands of users later, they shut it down.

I read it with some relief, because it confirmed a decision I keep making on AI products: one model from one provider. No router, no fallback chain, no complexity classifier. People occasionally tell me this is naive. Here is why the boring choice keeps winning.

The promise of routing is a spreadsheet, not a system

The pitch is always the same: simple questions are cheap, hard questions are expensive, so classify each request and pay accordingly. On a spreadsheet, this works beautifully. The savings compound and someone puts them on a slide.

The problem is that the classification happens at the worst possible moment: before the work starts. Manifest's core finding matches my experience exactly. You cannot judge complexity from the prompt alone, because complexity emerges while the work happens. In a research agent, "What does the policy say about early childhood learning?" looks simple. Then the search returns three overlapping document revisions from different years, and reconciling them is suddenly the hardest task of the day. A router already sent that question to the cheap model.

Consistency is the product

Take a citation-grounded assistant, the kind of product that promises every answer is backed by a source document, and that when the sources do not cover a question, the answer says so.

That behaviour is not a system prompt you paste in front of any model. It is the accumulated result of knowing exactly how one specific model handles retrieval context, when it drifts toward summarising instead of citing, what makes it refuse, and how it phrases uncertainty. Prompts, retrieval sizes, and output checks all get tuned against those exact behaviours.

Swap the model per request and every one of those calibrations becomes a guess. Two users ask the same question and get answers with different citation habits and different refusal thresholds. For a platform whose whole reason to exist is that it does not make things up, inconsistency is not a cost line. It is a product failure.

Caching killed the economics anyway

The financial case for routing quietly collapsed when prompt caching matured. Cached input tokens cost a fraction of uncached ones, but caching only pays when requests hit the same model with stable prompt prefixes.

A research agent is close to the ideal caching workload. The system prompt is long and fixed. The document context repeats across a conversation. Consecutive questions in one session share almost everything. Keep all of that on one model and the discount applies request after request:

const result = streamText({
  model: openai(MODEL_ID), // one model, pinned in config
  system: SYSTEM_PROMPT, // long, stable, cache-friendly
  messages: conversation,
})

Route the same session across models and you pay full price everywhere while adding a classifier to maintain. The router does not just fail to save money. It actively destroys the discount you already had.

The hidden cost is your evaluation budget

A small team gets a fixed amount of engineering attention, and every model in production spends some of it. Each one needs its own eval runs, its own regression suite, its own "why did this answer change" investigations.

With one model, an odd answer has a short list of suspects: the retrieval, the prompt, or the data. With a router, there is always a fourth suspect, and it is usually the one you check last at midnight. Manifest ran a real evaluation infrastructure and still concluded the unpredictability was not worth it. Most teams adopting routers have no such infrastructure. They will not measure the degradation. They will just feel it, one confused user at a time.

When a router actually earns its keep

To be fair, routing is not always the wrong call. It makes sense when the workload splits into genuinely separate task types known before execution: OCR cleanup here, long-form reasoning there. That is not one prompt classified on the fly; it is two features with two deliberate model choices. High-volume consumer products with millions of throwaway requests and a team dedicated to evals can also make the math work.

But that describes very few of the teams I see reaching for a router in week one, before they have even learned what their chosen model can do.

Choose boring, spend the savings on the product

The uncomfortable truth about most LLM cost optimisation is that the biggest win is rarely the model price. It is trimming bloated context, caching aggressively, retrieving less but better, and not sending the model work it does not need. All of those are easier with one model you understand deeply.

Pick a model. Learn its edges. Cache everything stable. Revisit the choice on a schedule, deliberately, with evals, instead of letting a classifier revisit it on every request. Your invoice will be fine, and your users will get the same product twice in a row. In this field right now, that consistency is a feature most products cannot offer.