AWS Introduces Ray Serve Deep Learning Container for Simplified GPU Inference

With TorchServe no longer actively maintained, AWS has introduced the Ray Serve Deep Learning Container (DLC) to address the challenges of managing GPU inference stacks. This container combines a tested, patched stack including the framework, GPU drivers, and serving layer, enabling developers to deploy models without worrying about compatibility or security issues.

The Ray Serve DLC is built on Amazon Linux 2023, with the GPU variant including NVIDIA's CUDA runtime libraries. It bundles PyTorch, Ray Serve with FastAPI and Uvicorn, and utilities for vision, audio, and multimodal workloads, ensuring all components are validated together.

This container supports deploying models like the Qwen3-VL vision-language model on Amazon EKS, with deployment scripts automating cluster setup, node provisioning, and Ray cluster deployment. The approach allows running inference on a single GPU node, with options for scaling to multi-node setups using KubeRay.

Developers can write their serving application as a Python class decorated with @serve.deployment, simplifying the process by removing the need for custom handlers or configuration files typical of TorchServe. The post provides an example of deploying a vision-language model that responds to HTTP requests with image descriptions.

The deployment process involves creating a ConfigMap with the application code, deploying the Ray Serve container on a GPU-enabled node, and verifying GPU utilization with nvidia-smi. Cleanup scripts are provided to dismantle the setup when no longer needed.

This new container streamlines the deployment of AI models in production, reducing maintenance overhead and improving security through regular patches. It also offers a clear migration path for teams currently using TorchServe, with support for scaling and multi-node serving via KubeRay.