AI Model Routing: 74% Cheaper, Six Accuracy Points Lost

Model routing has a headline number and a number nobody quotes. The headline: in a benchmark LangChain published on 11 August 2026, sending agent calls through a router instead of straight to a frontier model cut total cost by 74%. The number nobody quotes: it also cost six points of accuracy, dropping task success from 86.0% to 80.0%.

Both numbers come from the same table, and they only make sense together. This piece walks through what the benchmark actually measured, where the savings come from, and the three operational costs that decide whether routing is worth it for your workload.

The context

Model routing puts an arbiter in front of your LLM calls. For each request, that arbiter, usually called the judge or the router, decides whether it goes to a small fast model or to a slow expensive frontier one. Conceptually it is load balancing, with difficulty as the routing key instead of load.

What changed is agent economics. A chatbot makes one call per user message. An agent chains them: it reads, plans, calls a tool, reads the result, corrects itself. In the benchmark below, the average is 6.3 model calls per task. Your bill no longer tracks user count, it tracks reasoning steps. That is precisely the regime where routing starts to pay, because most of those 6.3 calls are mechanical: reformat this, extract that field, pick the next tool.

The bet behind routing is that difficulty is very unevenly distributed across an agent trajectory. The benchmark finally puts a number on that hunch.

What the Switchyard routing benchmark measured

The study, published by Srimanth Tangedipalli and Karan Singh, evaluates NVIDIA NeMo Switchyard, an open-source model-routing library. The protocol runs on the Deep Agents evaluation suite: 145 multi-step agentic tasks averaging those 6.3 model calls, spread across three realistic domains, customer support, incident investigation and multi-step automation.

Three arms are compared. The frontier model alone, Claude Opus 4.8. The small model alone, NVIDIA Nemotron 3.5 Lightning. And the routed arm, where a judge, Gemini 3.1 Flash Lite, arbitrates every call between the two.

Three arms, three bills

The middle row is the one that matters, and the article describes it plainly: routing was "74% cheaper and 6 points less accurate". Any pitch that quotes the first half without the second is selling you something.

Look at the bottom row too. The small model on its own is not embarrassing: 77.7% against 86.0% for the frontier. The whole spread between cheapest and best is only 8.3 points. Routing buys back a little over two thirds of that spread for roughly four times the small model's price. That is the trade you are making. It is not a free lunch.

93% of calls for 10% of the spend

The most useful figure in the study is not the total, it is the split. Nemotron handled 93% of model calls for 10.4% of the spend, while Opus handled 7% of calls for 68.4% of it.

If you operate agents, read that twice. It says that in a typical trajectory the overwhelming majority of calls are routine, and that a handful of decisive calls burn two thirds of the budget. It also says that if you currently bill every call at frontier rates, you are paying premium prices for work a 2024-vintage model would have handled fine.

The judge is not free

That leaves 21.2% of routed spend. That is the judge's share, and it is the line most teams forget to model. A router is a third model in production. It burns tokens on every single call, including the ones it ends up sending to the cheap model.

The article gives the break-even formula, and it is simple enough to fit in a spreadsheet: judge cost / (expensive cost - cheap cost). For this pairing, the system needed to offload 5.9% of turns for the judge to pay for itself. It offloaded 93%, clearing the bar by 16x. The margin is comfortable here, but it is not comfortable by construction. Move the cheap model closer to the expensive one in price and the denominator collapses. Routing between two similarly priced models never repays its judge.

The judge also costs time. The article puts its latency overhead at roughly 700ms per call. On a 6.3-call task that is over four seconds added to the full trajectory, best case, before any parallelism. For background automation that is invisible. For an agent answering a human who is waiting, it changes the category of experience you are shipping.

Variance is routing's blind spot

One detail deserves more attention than it usually gets. Across five runs, the share of traffic sent to the frontier model ranged from 4.1% to 9.1%, with a mean of 6.9%. Cost followed: the worst routed run came in at $3.61, about a third of Opus alone.

That spread falls straight out of the design. The judge is itself a probabilistic model, and how hard a trajectory turns out to be depends on what the tools return along the way. So you are not buying a cost. You are buying a cost distribution.

In practice, budget a routed system as a range, not a number. If your financial model rests on the mean you saw in a pilot, one week of hard traffic will blow through it without a single technical alert firing. Instrument frontier traffic share as a first-class production metric, with a threshold, the same way you would treat an error rate.

When routing is the wrong tool

The article is refreshingly explicit about where this applies. Routing targets teams that need frontier capability on hard requests and cannot tell in advance which ones those are. That is the load-bearing condition. If a reliable business rule can classify your requests up front, write the rule: it will be faster, free and deterministic.

Two cases to rule out. Latency-sensitive workloads, because of the judge's 700ms. And short workloads, where there are too few calls to amortise the added complexity: on a one or two call task you are adding a dependency and a failure mode to save a few cents.

What this changes for AI teams

The 74% makes the headline. Here is what actually matters once you have to operate the thing.

Six accuracy points is a product decision, not an engineering one. Going from 86% to 80% on agentic tasks means one task in five fails instead of one in seven. Depending on what your agent does, that is either painless or unacceptable. Set that floor with the people who answer customers, before you touch infrastructure. A cost-versus-quality trade made quietly inside the platform team always resurfaces as an escalation.

Instrument before you route. The break-even formula needs per-call cost for all three models and the share of offloaded calls. If your observability stops at a monthly total per API key, you cannot even compute whether routing is earning its keep. Per-call telemetry carrying model, cost, latency and outcome is a prerequisite, not a nice-to-have.

The judge is a production dependency in its own right. It gets versioned, monitored and rolled back like anything else. It also creates a new incident surface: quality can degrade because the judge started routing badly, with neither working model having changed. The day accuracy drops, your first question becomes "what moved in the routing split", and you need to answer it from a dashboard, not by grepping logs.

This fits a wider shift. LangChain is separately pushing managed agents, where runtime, state, sandboxes and observability ship together instead of being assembled by hand. Routing belongs to the same category: these are operations problems, not model problems. The same team also documented an autonomous SRE agent for Kubernetes deployments, another sign that agent work is moving out of the prototype and into the engine room.

Start by measuring your own distribution. Before installing anything, sample a hundred real trajectories and ask, call by call, whether a small model would have done. If the answer is yes 90% of the time, routing is for you. If it is yes 40% of the time, your problem is not routing, it is your prompt and tool design.

In short

• Across 145 agentic tasks, routing between Nemotron 3.5 Lightning and Claude Opus 4.8 was 74% cheaper and 6 points less accurate, 80.0% against 86.0%.
• The small model absorbed 93% of calls for 10.4% of spend; the frontier model took 7% of calls for 68.4%.
• The judge consumes 21.2% of routed spend and adds roughly 700ms per call: you need to offload at least 5.9% of turns before it pays for itself.
• Frontier traffic share ranged from 4.1% to 9.1% across five runs: budget a range, and monitor that ratio as a production metric.
• Skip routing if your workload is latency-sensitive, short, or already separable by a simple business rule.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Johannes Plenio on Unsplash.