MLOps

MoE batch inference: sort your prompts for 17% more throughput

MoE batch inference: sorting prompts by domain gives up to 17% more vLLM throughput. How it works, the real numbers, the limits and how to ship it safely.

MoE batch inference: sort your prompts for 17% more throughput

MoE batch inference: sort your prompts for 17% more throughput

Here is an uncomfortable question for anyone running MoE batch inference: does the order of your prompts cost you GPU hours? According to a study published on the Hugging Face blog on October 19, 2026, “Sort your prompts: a free speed-up for MoE batch inference”, the answer is yes. Group similar prompts together before launching a batch job and Qwen3-30B-A3B served by vLLM delivers up to 17.3% more end-to-end throughput. No new serving system, no model change, no predictor to train.

That is a rare kind of optimisation. It lives entirely in application code, needs no infrastructure change, and lands directly on the GPU bill for the jobs nobody watches closely: overnight evals, synthetic data generation, bulk document processing. The interesting part is not the headline number, though. It is understanding why it works, and the long list of cases where it does not.

The context

A Mixture-of-Experts (MoE) model does not use all of its parameters for every token. Each layer holds a set of “experts” (small sub-networks) and a router that sends every token to only a few of them. Qwen3-30B-A3B has 128 experts per layer across 48 layers; gpt-oss-120b has 32. That is how these models get large capacity at a manageable compute cost.

The catch sits in the decode phase, where text is generated one token at a time. Decode is bound by memory bandwidth, not by arithmetic: expert weights have to be streamed from HBM, the GPU's high-bandwidth memory. As the authors put it, every expert chosen by at least one token in the batch must be streamed from HBM once per layer. So the more distinct experts a batch touches, the more bandwidth you burn, even if a given expert only serves a single token.

Why prompt order drives MoE inference cost

Experts specialise by domain

The study starts from a routing observation. The authors built a set of 768 prompts across six domains: code (MBPP), maths (GSM8K), creative writing (no_robots), French, Arabic and Chinese. On Qwen3-30B-A3B, each domain concentrates its traffic on roughly 32 favourite experts, a quarter of the 128 available, which receive 68 to 80% of its routing decisions, versus 25% under uniform routing.

The consequence follows naturally. A batch made only of coding prompts touches fewer experts than one that mixes code, maths and Chinese. The measurements bear this out: on Qwen3-30B-A3B, single-domain batches read 19 to 25% fewer experts per step than mixed ones, at every batch size from 8 to 128. On gpt-oss-120b, whose fewer experts are necessarily more generalist, the gap shrinks to 8 to 11%.

A linear, predictable cost

The second observation is at the kernel level. Time spent in an MoE layer grows linearly with the number of distinct experts touched, at about 2.9 µs per extra expert, which corresponds to 9.4 MB of expert weights. On its own that is tiny. Multiplied by 48 layers and hundreds of decode steps per batch, it adds up to a real share of generation time.

What the benchmarks show

Tests ran on AMD Instinct MI300X GPUs with vLLM 0.27.1 (ROCm, CUDA graphs enabled), each prompt generating exactly 256 tokens with greedy decoding. The baseline is a shuffled order; the variant sorts prompts by domain.

On Qwen3-30B-A3B, throughput improves by 13.5% at batch size 16, 17.3% at 32, 12.6% at 64 and 11.6% at 128. On gpt-oss-120b, gains run from 6.9% at 16 to 12.2% at 32, then 10.6% and 9.1%. The post does not dwell on why 32 is the sweet spot, but the trend makes sense: at very large batch sizes even a homogeneous batch ends up hitting a large share of the experts, so there is less left to save.

Without domain labels

Production prompts do not arrive tagged “code” or “Chinese”. The authors try two ways to recover the groups:

  • Router fingerprints. After prefill, each prompt has a vector describing how often it used each expert in each layer. Running k-means on those vectors recovers the six domains with 100% purity and yields the same savings as the true labels. The downside: you need access to the router's internal decisions.
  • Text embeddings. Encode prompts with Qwen3-Embedding-0.6B, then run k-means. This needs nothing from the MoE model, runs before inference and works with any model.

On real conversational data (UltraChat), embedding-based clustering gives +5.1 to 5.5% on Qwen3-30B-A3B and under 4% on gpt-oss-120b. Smaller, but free. With natural output lengths (stopping at EOS, batch size 32), output throughput rises from 1,810 to 2,104 tokens per second, a 16.6% wall-clock speed-up in that setting.

Windowed sorting and expert parallelism

Two results matter for production pipelines. First, you do not need to sort the whole corpus at once. Sorting chunks of 128 prompts gives +11.7%, chunks of 256 give +14.8%, against +17.3% for a full sort of all 768 prompts. A reorder window of 4 to 8 batches keeps most of the gain, which makes the technique usable on streaming workloads. Second, with expert parallelism across two GPUs, the gain holds at +12.9% at batch size 64 and +8.7% at 128.

The limits to know before you sort your prompts

The post is candid about its blind spots, which is precisely what makes it useful.

Real gains are about half the theoretical ones. A model based purely on bytes read predicted +20 to 27% for Qwen. The gap comes from the rest of the pipeline: attention, the KV cache and sampling do not benefit from sorting.

Outputs change. Reordering changes batch composition, and therefore the numerics. The authors note that sorting changes outputs about as much as reshuffling does: 92% of outputs identical for Qwen against a shuffled baseline, and only 36% with natural output lengths. If your evals depend on bit-exact reproducibility, you need to pin the sorted order itself.

It is not universal. Dense models, prefill-heavy workloads, very large batches, coarse MoEs with few experts and latency-sensitive online serving see little or no benefit. In the online case, waiting for similar requests to accumulate adds queueing delay. The authors place the natural home elsewhere: offline batch jobs, or a scheduler that picks similar requests from its existing waiting queue.

Finally, run-to-run variance stays below 1% for Qwen but reaches 4% for gpt-oss, so on that model small gains deserve several runs before anyone puts them in a slide.

What this means for AI teams

For an MLOps team, the value of this study is less the 17% figure than the reminder behind it: MoE inference has a different cost profile from dense models, and part of that profile is controlled by the application, not just the serving stack.

Here is how we would approach it on a client engagement:

  1. Inventory your batch jobs. Nightly evals, synthetic data generation, ticket classification, document extraction: anything that runs offline on an MoE model is a candidate. Real-time serving is not.
  2. Measure before you sort. The gain depends on the model (number of experts), batch size and how heterogeneous your data is. A corpus that is already homogeneous, say code only, will gain little. Run the same job shuffled and sorted, several times, and compare tokens per second and total wall-clock time.
  3. Start with embeddings. It is the easiest method to wire in: a small embedding model, k-means, a stable sort, then restoring the original order. The authors' repository ships routing capture, a simulator, a Triton kernel and the vLLM benchmarks. Usage fits in three lines:
from sort_prompts import expert_friendly_order

order = expert_friendly_order(prompts, n_clusters=16)   # embed + k-means + stable argsort
outs = llm.generate([prompts[i] for i in order], sampling_params)
outs = [o for _, o in sorted(zip(order, outs))]          # restore original order
  1. Make reproducibility an explicit requirement. If a job is an evaluation baseline, version the prompt order alongside the dataset. Otherwise a throughput win can turn into results that drift between runs with nobody able to explain why.
  2. Think in windows. For a streaming pipeline, a window of a few batches recovers most of the gain. Expose it as a configuration parameter, just like batch size.

On a GPU fleet billed by the hour, 10 to 15% more throughput on batch jobs is machine time handed back, without new hardware or a different model. It is also a good excuse to add observability to batch jobs that are often launched blind: tokens per second, time per batch, cost per thousand requests.

Key takeaways

  • During decode, an MoE model pays in memory bandwidth for every distinct expert the batch touches.
  • Sorting prompts by domain cuts that number: up to +17.3% throughput on Qwen3-30B-A3B and +12.2% on gpt-oss-120b with vLLM on MI300X.
  • Without labels, embedding-based clustering gives roughly +5% on real conversational data, with no change to the model.
  • Outputs shift about as much as with a reshuffle, and the technique does nothing for latency-sensitive online serving.
  • For MoE batch inference, this is an application-level optimisation worth testing first on your offline jobs.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Jan Antonin Kolar on Unsplash.