Reducing ASR Inference Costs with NVIDIA MPS on Amazon EC2

Serving automatic speech recognition (ASR) models at scale can be costly due to low GPU utilization per request, despite strict latency requirements. A collaboration between AWS, NVIDIA, and Heidi Health demonstrates how NVIDIA CUDA Multi-Process Service (MPS) combined with Triton Inference Server on Amazon EC2 GPU instances can address this challenge.

Typically, a single ASR inference request utilizes only about 15-20% of a GPU’s compute capacity, leaving the majority of the hardware idle during each forward pass. The default CUDA time-slicing behavior enforces sequential access, which results in underutilization and limits throughput. Heidi Health, processing over 2.4 million consultations weekly, currently deploys 16 GPU instances to meet latency SLAs.

The solution involves leveraging CUDA MPS, which allows multiple processes to share a GPU concurrently, eliminating context-switching overhead and enabling kernels from different processes to run simultaneously. This setup partitions the GPU into multiple concurrent instances, significantly increasing utilization.

Three GPU sharing mechanisms are evaluated: default time-slicing, Multi-Instance GPU (MIG), and CUDA MPS. The latter supports concurrent kernel execution without code changes, providing a flexible and efficient sharing method. By configuring MPS with appropriate environment variables, the system supports multiple concurrent processes, each with dedicated SM allocations.

Further optimization is achieved through model-level enhancements using ONNX Runtime with TensorRT, which applies kernel fusion, precision calibration, and memory optimization to accelerate the compute-heavy encoder. The inference pipeline employs Triton Inference Server for request batching and scheduling, supporting dynamic batching for transcription and sequence batching for diarization.

The architecture runs on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances, orchestrated via Docker Compose. The pipeline includes a FastAPI gateway for audio decoding, Triton for inference management, and CUDA MPS for GPU partitioning. The deployment process involves building container images with the fine-tuned model, configuring MPS instances, and running the inference services.

Benchmark results show that using CUDA MPS reduces the number of GPU instances needed from 16 to 4 while maintaining sub-second latency at high request rates. Specifically, on g7e.4xlarge, the system achieves 92.1 requests per second per GPU at a mean latency of 352 ms. Further optimization with TensorRT and ONNX increases throughput to 111.6 RPS, representing an 88% reduction in infrastructure.

The approach extends to speaker diarization models, which also benefit from CUDA graph warmup optimizations, achieving predictable latency and real-time processing capabilities.

In conclusion, NVIDIA CUDA MPS on Amazon EC2 enables substantial reductions in ASR inference infrastructure costs while meeting latency SLAs. The techniques are model-agnostic and applicable to workloads where individual requests utilize small GPU fractions. The accompanying open-source repository provides the complete deployment setup, including Dockerfiles, Triton configurations, and API code, facilitating adoption and customization.