Your Router Might Be a Constant
1. The Bill, Again
In my previous post on LLM cost and latency, I mentioned three levers for keeping inference bills under control: caching, batching, and routing. Routing was the one with the asterisk—"Dynamic Model Routing: cheap model for easy questions, premium for hard ones. Gotcha: must detect 'hardness' reliably."
This post is that gotcha, fully unpacked.
Because it turns out "detecting hardness reliably" is the entire problem, and most teams aren't measuring whether they're solving it at all.
2. The Five Things People Mean By "Routing"
The term is badly overloaded. When someone says "we're routing LLM requests," they could mean any of these:
Technique How it decides Overhead Real gotcha
------------------- ---------------------------------- ------------ ---------------------------------
Semantic Embed query, cosine-match to 20–50 ms Reference-prompt curation is
reference prompts the whole job; nobody budgets it
Cost-aware Classifier predicts difficulty, ~classifier Needs preference-labelled data
picks cheapest adequate tier you probably don't have
Intent-based Heuristic cascade → light µs → ms Multiple passes; latency variance
classifier only when ambiguous
Cascading Run cheap, score confidence, 1–2× calls Unbounded p99. Bad fit for
escalate on failure anything latency-sensitive
Load balancing Health/weight across providers ~0 Not routing. It's plumbing—
and keys but it's the plumbing to build first
Only the middle three are "routing" in the sense people mean when they talk about cost savings. The last one—load balancing—is reliability infrastructure and should be your foundation, not your differentiator.
The first four all share the same hidden assumption: the router's decisions vary meaningfully with the input, and that variance produces better outcomes than just picking one tier and sticking with it.
Most teams measure neither half of that assumption.
3. The Result That Should Change Your Default
On July 28, 2026, Kumar and Saminathan published a paper (arXiv 2608.14641) that evaluated four open-source routers—RouteLLM, LiteLLM, vLLM Semantic Router, and Aurelio—across four benchmarks: RouterBench, Berkeley Function Calling Leaderboard (BFCL) v4, tau2-bench, and WebArena. 290 tasks, 2,610 candidate outcomes.
The headline finding: three of the four routers emitted static or near-static tier assignments. Only vLLM Semantic Router showed material variation with prompt content—and it still didn't top any of the four benchmarks.
Here are the tier success rates they measured:
Benchmark Cheap Mid Strong Note
----------- ------- ------- -------- -----------------------------------------
RouterBench 0.300 0.683 0.800 Clean monotonic tier ordering
BFCL v4 0.833 0.811 0.800 INVERTED—cheap model wins function calls
tau2-bench 0.543 0.810 n/a Multi-turn retail agent
WebArena 0.110 0.220 n/a Web nav; everything is bad, mid is 2× cheap
Two numbers worth quoting directly:
- "Always-Mid matches Aurelio exactly on three benchmarks and within 0.003 on the fourth."
- No statistically significant task-specific success advantage on any benchmark. Gains tracked the selected-tier composition, not discriminative routing.
In other words: the routers that showed cost savings were saving money because they were sending more traffic to cheaper tiers, not because they were making smart per-request decisions.
The BFCL Inversion
Look at that second row again. On function calling, the cheap model (0.833) beat the strong one (0.800). A router optimized to send "hard" requests to the strong tier is actively making things worse.
Tier ordering is not a law of nature. It's a per-task empirical fact.
If you assume GPT-4 beats GPT-3.5 on everything, you will route badly on the tasks where that's false. And you won't know unless you measure the fixed-tier baselines first.
The Honest Caveat
The authors scope their findings to the tested configurations and frozen benchmark samples. This is a warning about evaluation practice, not proof that routing can never work. But it's a loud warning, and it comes with receipts.
4. What a Router That Actually Routes Looks Like
So what does a router that does vary with input look like? Let's talk about the one I actually work on: vLLM Semantic Router.
Full disclosure: it was the one router in the Kumar & Saminathan study that showed prompt-dependent variation. It also didn't win any of the four benchmarks. I'm going to state that plainly rather than gloss over it, because the interesting part isn't "we won"—it's the architecture that made variation possible in the first place.
Signals and Decisions Are Separate Layers
The core design idea worth stealing, even if you never install the project:
Client → Envoy (ext_proc, gRPC) → Go ExtProc router → backend model
│
signals ────┴──── decisions
(13 signal types) (Boolean rules → model choice)
Signals extract properties from the request: language, complexity, modality, safety flags, PII, jailbreak attempts, domain classification. They're implemented as classifiers, embeddings, regex, BM25, n-gram fuzzy match—whatever fits the signal.
Decisions are Boolean rules matched against those signals. "If language is not English AND complexity is high, route to multilingual-strong. If PII detected, route to local-only tier."
You can swap a classifier without rewriting policy. You can read the policy without reading model weights. When a request routes somewhere surprising, you can see which signal fired.
This is the opposite of an end-to-end learned router, where the decision function is a black box. The tradeoff: you have to curate the signals and write the rules. That's engineering work, not just training work.
What's Actually Running
The project has gone through a few iterations. Here's what's in the current release (Athena, v0.2):
Embedding model: mmBERT-Embed-32K 2D-Matryoshka, 307M parameters, 1800+ languages. It's a 768-dimensional model that can be truncated to 256 dimensions at roughly 99% quality retention. That truncation matters for latency.
Classifiers: Eight of them in the mom-multilingual-class family—intent, jailbreak, PII detection, fact-check, feedback, and a few others. They're merged models with LoRA variants for different domains.
Deterministic signals: BM25 for keyword routing, n-gram fuzzy match, regex. These are microsecond-scale and less brittle than pure embeddings when you have known trigger phrases.
Guardrails: HaluGate (3-stage hallucination detection) and ReflectionGate (episodic memory for multi-turn conversations). These aren't routing signals—they're quality gates that can block or rewrite requests before they hit the model.
Runtime: Rust/Candle core for the heavy lifting, Go for the Envoy ExtProc integration, ONNX + CK Flash Attention for GPU inference.
The Latency Numbers
This is the number that decides whether routing is viable at all. The project reports these figures (measured on AMD MI300X hardware, which is important context):
Path Latency Source
------------------------------------ ----------- -----------------------------
Domain extraction, CPU 630 ms Athena release notes
Domain extraction, GPU (MI300X) 10.2 ms Athena release notes
3 classifiers, CPU (~500 tok) 853 ms Athena release notes
3 classifiers, ONNX+GPU (~500 tok) 22 ms Athena release notes
End-to-end routing @ ~16K tok 103 ms (was 143 ms pre-CK-FA)
P50 routing overhead vs. inference 0.4%–5% vs. 800–11,000 ms inference
The honest framing: on CPU, classification costs more than it saves for short prompts. 853 ms of routing overhead to save 800 ms of inference time is a net loss. That's a real deployment constraint, and almost nobody writing about routing mentions it.
On GPU, the math flips. 22 ms to potentially save thousands of milliseconds (and dollars) on a strong-tier call is a clear win—if the router actually routes correctly.
The Token-Optimization Run
In one of the getting-started examples, the project routed 86% of 21 test prompts to a free local model. Small n—I'm saying so explicitly—but it's the shape of the win when routing works: most requests are easy, and you only pay for the hard ones.
The catch: you need to have measured that "most requests are easy" is true for your workload, not someone else's benchmark.
5. Agents Are Where This Actually Pays
If you've read my post on OpenClaw, you know I'm interested in agent systems that run locally and handle repetitive workflows. Agent loops are the ideal routing workload:
- High volume: Agents make dozens to hundreds of LLM calls per session.
- Repetitive: Planning, tool selection, result parsing—many of these are structurally similar.
- Mostly easy: The majority of agent calls are not "write a novel" or "prove this theorem." They're "parse this JSON" or "summarize these three lines."
The Athena release notes specifically call out OpenClaw as a target integration: OpenAI-compatible endpoint, no code changes required, router picks the model per request based on the signals it sees.
There's also ClawOS, which orchestrates multiple OpenClaw agent systems under one router with shared memory and guardrails. I'm flagging it as experimental—it's not production-ready—but the architecture is interesting if you're building multi-agent systems.
The key insight: routing pays when you have a distribution of request difficulties, not when every request is equally hard. One-shot chat is the worst case for routing. Agent loops are the best case.
6. The Code: Prove Your Router Isn't a Constant
Here's a short Python function you can run against your own router to check whether it's actually varying with input:
from collections import Counter
import math
def routing_report(router, queries):
"""Does your router actually vary with input, and does it beat a constant?"""
picks = [router(q) for q in queries]
dist = Counter(picks)
n = len(picks)
# Shannon entropy of the tier distribution: 0.0 == you shipped a constant
H = abs(-sum((c/n) * math.log2(c/n) for c in dist.values())) # abs() kills -0.0
modal_share = dist.most_common(1)[0][1] / n
print(f"tiers used : {dict(dist)}")
print(f"entropy : {H:.3f} bits (0.0 = constant router)")
print(f"modal share: {modal_share:.1%} (>0.95 = effectively constant)")
return H, modal_share
The rule: If entropy is near zero, delete the router and hardcode the modal tier. You'll get the same quality, lower p99 latency, and one fewer thing to page on.
If entropy is high, you still have to beat Always-Cheap and Always-Mid on your own eval set before you've earned the complexity.
This is a diagnostic, not a proof. But it's a diagnostic almost nobody runs, and it catches the failure mode the paper found: routers that look sophisticated but behave like constants.
7. Checklist Before You Route
Before you add a router to your stack, measure these things:
-
Fixed-tier baselines measured first. Run Always-Cheap, Always-Mid, Always-Strong on your own eval set. These are the numbers to beat.
-
Per-task tier ordering verified. Don't assume strong > cheap. BFCL says otherwise for function calling. Measure it.
-
Routing entropy logged in production, not just at eval time. Distributions drift. If your router starts collapsing to a constant six months in, you want to know.
-
Routing overhead measured against your p50 inference, on the hardware you'll actually deploy on. CPU classification changes the math completely.
-
p99 bounded. Cascading routers have unbounded escalation paths. Cap the hops, or accept that your p99 is "however long it takes to fail up to the strongest tier."
-
Fallback path when the classifier is unavailable. Default to a tier, don't throw an error.
-
Policy readable without reading weights. Signal/decision separation, or you can't audit why a request went where it went.
If you can't check all seven, you're not ready to route. Build the plumbing first.
8. Final Thought
The best routing result of 2026 so far is a null result.
Every cost-savings number you read is a claim about a distribution of queries, not about a router. And until you've measured your own constant baseline, you don't know which one you bought.
The vendors will keep publishing "85% cost reduction" benchmarks. Some of them are real. Some of them are just "we sent 85% of traffic to the cheap tier, and the cheap tier was good enough." Those are not the same thing.
The difference is entropy. Measure it.
9. What's Next
If you're building with LLM routing, here are two concrete next steps:
Try the entropy probe. Run the code from section 6 against your own traffic. If it comes back near-zero, you've saved yourself from deploying a constant. If it's high, you still need to beat the fixed-tier baselines.
Measure your own tier ordering. Don't assume expensive models beat cheap ones on every task. The BFCL result showed the opposite for function calling. Run Always-Cheap, Always-Mid, and Always-Strong on your actual workload before you route.
If you want to experiment with routing, vLLM Semantic Router is Apache-licensed and OpenAI-compatible. The quickstart takes about 15 minutes.
If this was useful, let me know on LinkedIn or subscribe for more posts on ML systems, cost optimization, and agent infrastructure.