Reduce inference cold starts on Amazon SageMaker HyperPod with model caching

Amazon SageMaker HyperPod has introduced a new feature called model caching, designed to address the cold start problem in deploying large language models (LLMs) for inference. When deploying models, especially large ones like DeepSeek-R1, the startup process involves downloading container images and model weights, which can take tens of minutes, hindering rapid scaling and responsiveness.

Model caching pre-loads these components onto local NVMe storage on cluster nodes before pods are scheduled. This process involves two capabilities: weights cache, which downloads model weights to local storage, and image cache, which pre-pulls container images onto nodes. When a pod starts, it reads from local storage at high speeds, significantly reducing startup latency.

Enabling these caches involves adding specific configurations to the deployment resources, with no additional infrastructure setup required. The process is managed automatically by the SageMaker HyperPod operator, which handles cache lifecycle, updates, and cleanup.

Benchmarks indicate that enabling weights caching can accelerate scale-out times by approximately 60%, and image caching can reduce image pull times by up to 97%. These improvements are especially impactful for models exceeding hundreds of gigabytes, where startup times previously could reach over 30 minutes.

The caching is designed to be resilient, with fallback to standard download procedures if caches are not available or during initial population. The feature supports various model sources, including Amazon S3, FSx for Lustre, HuggingFace Hub, and SageMaker JumpStart, across all supported instance types with sufficient NVMe storage.

This feature is now generally available in all regions supporting SageMaker HyperPod, offering a practical solution to improve deployment efficiency for large AI models.