MLOps

NVIDIA MPS on EC2: Cutting GPU Inference Infra by 75%

NVIDIA MPS lets Heidi Health cut ASR inference GPU infrastructure by 75% on Amazon EC2, with zero application code changes — here's how it works.

NVIDIA MPS on EC2: Cutting GPU Inference Infra by 75%

NVIDIA MPS on EC2: Cutting GPU Inference Infra by 75%

NVIDIA MPS can cut the number of GPUs required to serve an inference model by four — without touching a single line of application code. If you've ever watched a production GPU dashboard and wondered why utilization sits stubbornly around 20%, you already know the problem this solves. A technical post published August 27, 2026 by AWS and NVIDIA, co-written with Heidi Health, walks through exactly how NVIDIA MPS (Multi-Process Service) turned that idle capacity into a 75% infrastructure reduction in a real production speech-recognition (ASR) pipeline.

Heidi Health processes 2.4 million clinical consultations per week across 190 countries. At that scale, a 75% drop in GPU count isn't a minor tuning win — it's a structural change to the infrastructure bill.

The problem: idle silicon by design

A single ASR inference request typically uses only 15-20% of the streaming multiprocessors (SMs) on a GPU like the NVIDIA L40S, which has 142 of them. By default, CUDA shares a GPU across processes through time-slicing: each process gets exclusive, sequential access, taking turns. The GPU spends most of its time waiting, and context-switching overhead compounds the waste on top of that.

NVIDIA CUDA MPS is a binary-compatible alternative implementation of the CUDA API that lets multiple processes share a GPU through genuinely concurrent execution — running on different SMs of the same GPU at the same time, instead of taking turns. Unlike MIG (Multi-Instance GPU), which physically partitions the hardware, MPS is purely a software layer and can be reconfigured dynamically.

How MPS actually works

The core mechanism: MPS routes all CUDA work through a single GPU context managed by an MPS daemon process. That eliminates most of the context-switching overhead and allows kernels from different processes to execute concurrently on the available SMs. Memory protection is preserved — each process keeps its own address space.

Partition size per instance is configurable through the CUDA_MPS_ACTIVE_THREAD_PERCENTAGE environment variable. In Heidi Health's deployment, transcription runs on 4 concurrent MPS instances at 25% SM allocation each (roughly 2.5 GB VRAM per instance), while diarization (speaker identification) runs on 8 instances at 12% each (roughly 1.8 GB VRAM). The two workloads are deliberately kept on separate MPS partitions to avoid contention.

The full pipeline

Three components make up the architecture: a FastAPI gateway compatible with the OpenAI Whisper API (audio decoding via torchcodec, port 8002, 4 uvicorn workers) forwards requests over gRPC to an NVIDIA Triton Inference Server. Triton handles dynamic batching for transcription and sequence batching for diarization, with preferred batch sizes of 4, 8, and 16, and a 50ms max queue delay. The CUDA MPS daemon partitions the GPU before Triton even initializes.

The model is NVIDIA Parakeet TDT 0.6B V2, fine-tuned for clinical speech recognition (a 24-layer Conformer encoder with an RNN-T decoder). Two further optimizations trim latency: an ONNX Runtime + TensorRT path applying kernel fusion and precision calibration (FP16/INT8), and calling model.forward() directly instead of model.transcribe() through NeMo — which removes roughly 50ms of framework overhead per request.

MPS vs. MIG: picking the right tool

NVIDIA offers two distinct GPU-sharing mechanisms, and they solve different problems. MIG (Multi-Instance GPU) physically partitions a compatible GPU (A100, H100, and newer generations) into fully isolated instances with dedicated memory and compute — maximum isolation, but fixed granularity and a hardware-limited instance count. MPS, by contrast, stays software-defined: partition sizes are configurable on the fly via an environment variable, no hardware restart required, and it works on a broader range of GPUs — including the L40S used here, which doesn't support MIG at all.

The trade-off: MPS shares a single GPU context across processes, which in theory exposes cross-contamination risk if a neighboring process crashes — hence the CUDA Graph safety mechanisms and health-check sentinel described below. For an internal ASR service where every process runs the same model under one team's control, that trade-off is generally acceptable. For strict multi-tenant workloads running untrusted code, MIG remains the safer default.

The numbers

Benchmarks were run on g6e.4xlarge and g7e.4xlarge instances (NVIDIA L40S GPU, 48GB VRAM), against an SLA of mean latency under 650ms and p99 under 1,000ms:

  • Triton + MPS on g6e.4xlarge: 60.8 requests/second (RPS) per GPU, 470ms mean latency, 947ms p99 — 75% infrastructure reduction (16 GPUs down to 4).
  • Triton + MPS on g7e.4xlarge (the production path): 92.1 RPS per GPU, 352ms mean latency, 769ms p99 — same 75% reduction, with 51% higher throughput and 25% lower latency than the g6e configuration.
  • TensorRT + ONNX + MPS on g7e.4xlarge: 111.6 RPS per GPU, 590ms mean latency, 896ms p99 — an 88% infrastructure reduction (16 GPUs down to just 2).

Without MPS, a single L40S under default time-slicing caps out around 62 RPS, with 80% of its SMs sitting idle during each forward pass. Put differently: moving from the naive time-sliced baseline to the TensorRT+ONNX+MPS configuration is roughly a 1.8x throughput gain per GPU on top of the 88% fleet-size reduction — the two effects compound rather than trade off against each other, which is what makes the economics here different from a typical "batch size tuning" optimization.

On the diarization side, adding a TensorRT warmup cut mean latency from 309ms to 239ms (-23%) and p99 from 499ms to 389ms (-22%). A 60-second recording, split into four 15-second chunks, processes at a real-time factor of 0.016x.

The engineering it takes to keep NVIDIA MPS stable

The authors are upfront about what the headline number doesn't show:

  • Serialized model loading: a file lock prevents multiple instances of a 600M-parameter model from loading simultaneously, which would otherwise blow past available GPU memory.
  • CUDA Graph warmup envelope: expected production shapes (5, 15, 30, 45, 60 seconds at batch size 1, plus batch size 2 at 61 seconds) are pre-warmed to replay cached graphs at roughly 165ms. Any shape outside that envelope falls back to eager execution at roughly 500ms — a 3x latency penalty.
  • MPS-safe CUDA Graph fallback: without it, a 1.5-2.5 second recapture window can let a neighboring MPS instance corrupt the capture, triggering cudaErrorIllegalAddress.
  • Sentinel health monitoring: a dedicated mechanism detects unrecoverable CUDA errors on an MPS instance via a CUDA stream probe, writing a sentinel file to tmpfs to trigger instance replacement.

The authors state the architecture — MPS, direct forward-pass calls, CUDA Graph safety, health sentinel — is model-agnostic: it's been validated on Parakeet TDT, Canary, and OpenAI Whisper large-v3, and applies in principle to any encoder-decoder model served through Triton where each request only needs a fraction of the GPU.

What this means for AI teams

This post highlights a failure mode a lot of MLOps teams don't measure closely enough: over-provisioning disguised as capacity planning. When a single inference request only uses 15-20% of a GPU, adding more GPUs to absorb load just means paying for idle silicon. MPS isn't a niche trick — it's a service-architecture change that can cut a GPU infrastructure bill by four, or by eight with ONNX/TensorRT layered on top, without touching the model's business logic.

The real cost isn't turning MPS on — that's an environment variable. It's the reliability engineering around it: managing concurrent loading, warmup envelopes, CUDA Graph corruption detection, dedicated health checks. That's exactly the kind of work — profiling actual GPU utilization, right-sizing inference infrastructure accordingly, and industrializing the reliability layer that comes with it — that a production AI infrastructure audit is built to surface.

A practical way to check whether this applies to your stack: measure real per-request SM utilization on your current production load (via nvidia-smi or metrics you already collect in CloudWatch/Prometheus); check whether your workloads run the same model under one team's control (favoring MPS) versus untrusted multi-tenant code (favoring MIG); then prototype the MPS configuration on a slice of traffic before a full rollout, treating latency SLA as the guardrail rather than raw throughput alone. That's a multi-week engineering effort, not an afternoon config change — but based on the Heidi Health numbers, the payoff shows up directly as a GPU bill cut by four.

For any team serving inference at volume — ASR, but the same pattern applies to embeddings, classification, or lightweight vision models — the question worth asking today: what percentage of your production GPUs is actually busy during a forward pass? If the answer is close to 20%, NVIDIA MPS is probably the most under-used cost lever in your stack.

Key takeaways

  • NVIDIA MPS enables genuinely concurrent GPU execution across processes, unlike CUDA's default time-slicing.
  • Heidi Health cut its ASR inference GPU footprint by 75% (16 → 4 GPUs) while holding a sub-second latency SLA.
  • Layering TensorRT + ONNX on top of MPS pushes the reduction to 88% (16 → 2 GPUs).
  • The gain comes with real engineering cost: serialized loading, CUDA Graph warmup, corruption detection, dedicated monitoring.
  • The pattern is model-agnostic and applies to any inference workload that under-utilizes the GPU per request.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Kevin Ache on Unsplash.