MLOps

A/B Testing Models in Production Beats Shadow Traffic Alone

Shadow traffic proves an AI model is operationally stable, not that users actually prefer it. Here's how to run A/B testing at the endpoint level in production.

A/B Testing Models in Production Beats Shadow Traffic Alone

A/B Testing Models in Production Beats Shadow Traffic Alone

A/B testing in production is what your shadow traffic setup is quietly missing. Your dashboard is green — latency fine, error rate fine, cost per request fine — so why do support tickets keep creeping up since you swapped in the new model? Together AI's August 17 post names the gap directly: shadow traffic proves a candidate is operationally sound, but it can't tell you if users actually like it better. For any team that regularly rotates model versions — a new Claude generation, a Mistral update, a freshly fine-tuned internal model — that gap is exactly where regressions hide.

The background

Shadow traffic duplicates a slice of production requests to a candidate model without ever showing its output to the real user. The user keeps seeing the incumbent's response; behind the scenes, the candidate processes the same request and gets compared on operational metrics — latency, error rate, cost per call, stability under load. It's a low-risk practice: no chance of a buggy candidate ever reaching a real user.

That safety comes at a cost. Shadow traffic never measures preference. A model can match the old one on speed, reliability, and cost, while quietly producing answers that are less useful, less accurate, or worse-phrased — defects that show up in no infrastructure metric, only in how real users behave when they see a real answer.

Why shadow traffic alone isn't enough

Together AI puts it in one blunt line: "Shadow traffic proves a candidate is operationally sound. It can't tell you if users like it better."

Teams tend to discover this gap at the worst possible time — after cutting over to a new model purely on infrastructure metrics, then noticing a quiet rise in re-prompts, support tickets, or session abandonment, with no technical alert ever firing. The model was never "down." It was just worse, in a way no latency dashboard was built to catch.

Running A/B testing in production at the endpoint, not in application code

Together AI's proposal is to move the A/B split out of application code and into the endpoint infrastructure itself. Instead of an if/else in the service that calls the model, the traffic split between variant A and variant B is handled by the inference layer.

That's not just tidier code. A split managed at the infrastructure level lets teams change the ratio — 10%, then 25%, then 50% — without touching application code or redeploying anything. It also centralizes measurement: both variants pass through the same observability point, avoiding the bias that creeps in when different services each implement their own ad-hoc split.

Together AI isn't alone in this pattern. Google Cloud documents a comparable mechanism on Vertex AI, splitting traffic between model versions during a rolling deployment so traffic migrates in stages rather than an abrupt cutover. The underlying idea shows up across inference platforms: decouple the routing decision from business logic.

Keeping cohorts stable across a session

One technical detail teams often underestimate: the same user needs to see the same variant for the entire test window. Without consistent routing by user or session, a single person might get model A's response one turn and model B's the next — which makes any preference measurement meaningless, since you're no longer comparing two distinct populations, just noise. The common fix is hashing a stable identifier (user, session, or account) to assign a cohort deterministically and keep it stable for the duration of the test.

Picture the alternative: a team ships a cheaper, faster model behind a feature flag scattered across three services, watches infrastructure dashboards stay green for two weeks, and only learns something's wrong when a customer success lead flags a spike in "this isn't what I asked for" tickets. Rolling the split back out means finding every flag, in every service, and hoping none were missed. An endpoint-level split collapses that into one dial, in one place, that can be turned back to zero in seconds.

What to measure beyond latency and error rate

Once the split is in place, the next question is what to actually measure. Infrastructure metrics — latency, error rate, cost — remain necessary; a candidate that degrades any of them should be ruled out before preference even enters the conversation. But they don't settle the question between two equally stable models. What's missing are real satisfaction signals:

  • Re-prompt rate — a user who immediately rephrases their question is often signaling an unsatisfying first answer.
  • Resolution rate — did the task or conversation end without a human stepping in?
  • Explicit feedback — thumbs up/down, ratings, or any directly collected satisfaction signal.
  • Downstream behavior — session abandonment after a candidate-model response, compared against the baseline variant.

One discipline worth not skipping: these signals are noisy, and concluding too early on a small sample is a confident way to be wrong. A test window long enough, and a sample size calculated in advance rather than eyeballed afterward, is what separates a real A/B test from a gut feeling.

Knowing when to call it

A badly-ended A/B test is almost as misleading as one that never ran. Two mistakes show up constantly. The first: stopping as soon as a favorable gap appears, before reaching the pre-calculated sample size — a reliable way to generalize pure noise. The second, quieter one: letting the test run indefinitely "to be safe," which delays a decision the added certainty doesn't actually justify, given the cost of running two models in parallel.

Good practice fixes three things before the test starts: the size of the gap that would justify switching (say, a 5-point gain in resolution rate), the sample size needed to detect that gap with reasonable confidence, and a date or traffic volume beyond which the test ends regardless — winner declared or not. That last rule keeps a test with too wide a margin of error from staying "still evaluating" forever without ever producing a decision.

There also needs to be a fast exit in the other direction: if the candidate degrades an operational signal mid-test — not preference, but latency or error rate — the test should be killable immediately, without waiting for the planned window to close. A solid production A/B testing setup builds its own early-stop conditions in from the start.

What this changes for AI teams

For a team industrializing AI agents or assistants in production, this shift in method has a concrete consequence: every model version upgrade stops being a bet and becomes a measured decision. Instead of choosing between "keep the old model out of caution" and "cut everyone over and hope," the team gets a middle path — expose a controlled slice of traffic, measure real preference, and only roll out wider once the signal is clear.

That matters more as the generative model layer gets treated as swappable rather than fixed — routing to whichever model fits a given request type, weighing cost against quality per use case, keeping an eye on each new model generation as it ships. A clean endpoint-level A/B test is what makes that continuous replacement something a team steers, rather than something that happens to them.

It also changes how a model migration gets pitched internally. Instead of an argument built on public benchmarks or infrastructure dashboards, the team can show a preference signal measured on its own real traffic, with its own users and its own use cases — a far sturdier case in front of stakeholders who need to sign off on the change.

There's one more payoff that's easy to undersell when justifying the infrastructure investment: once the traffic split and preference measurement exist for one test, the same setup serves far more than model-version comparisons. It becomes the default rig for testing a new system prompt, a new retrieval strategy in a RAG pipeline, or a new tool set exposed to an agent. The discipline — stable cohorts, a pre-set decision threshold, early-stop conditions — carries over as-is, well past the original model-swap use case. It's infrastructure that pays for itself across the full production lifecycle of an AI system, not just one migration.

In short

  • Shadow traffic proves a candidate model is operationally sound, not that users prefer it — two different questions.
  • Splitting traffic at the endpoint instead of in application code lets teams adjust ratios without redeploying, and centralizes measurement.
  • Routing must stay consistent per user or session for the whole test window, or you're measuring a blend instead of two distinct populations.
  • Preference metrics (re-prompts, resolution, explicit feedback) complement infrastructure metrics — neither one alone is enough.
  • A pre-set test window and sample size keep teams from calling a noisy signal too early.

Done right, A/B testing in production becomes permanent infrastructure for any decision touching an AI system's behavior — not a one-off rig reserved for model swaps.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Taylor Vick on Unsplash.