Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

On August 12, 2026, Alibaba’s Qwen team released Qwen3.8-2.4T-A95B, a large open-weight AI model with 2.4 trillion parameters designed for demanding reasoning and agentic workloads. This marks the first time a Qwen-Max-class model has been made available as open weights, offering full control over inference behavior and data privacy.

Deploying such a massive model requires purpose-built infrastructure and optimized serving stacks. Amazon SageMaker HyperPod, combined with vLLM, provides a scalable solution for hosting models of this size. The deployment process involves cluster provisioning, NVFP4 quantization to reduce memory footprint, and configuring an OpenAI-compatible endpoint with features like reasoning, tool calling, and native Multi-Token Prediction (MTP) speculative decoding.

Qwen3.8-2.4T-A95B features a hybrid attention architecture with 92 layers, including gated DeltaNet and full quadratic attention layers, enabling efficient long-context inference up to 262K tokens native, extensible to over 1 million tokens. Its design supports complex reasoning, multi-step coding, and autonomous tool use, with built-in controls for reasoning depth.

The open weights are available on Hugging Face, with community quantizations such as MXFP4 and NVFP4 (W4A4), which compress the model to approximately 1.2 TB, fitting within the GPU memory of an 8-GPU node. Benchmark results indicate strong performance in research workflows, instruction following, and coding tasks, positioning Qwen3.8 as a credible frontier model for organizations seeking self-hosted alternatives.

Deploying on AWS involves creating a HyperPod cluster orchestrated by Amazon EKS, with the HyperPod Inference Operator managing model download, container scheduling, health monitoring, autoscaling, and node failures. The recommended instance type, ml.p6-b300, provides 8 NVIDIA B300 Ultra GPUs with 288 GB of GPU memory each, supporting the model’s requirements.

To handle the model’s size, NVFP4 quantization reduces the weight footprint to about 1.2 TB, fitting within the GPU memory. The deployment process includes preparing a YAML manifest specifying the model, container image, and resource requests, then applying it via kubectl. The process involves model download, weight loading, health checks, and endpoint readiness, typically taking 15-30 minutes on a fresh deployment.

Once deployed, the endpoint offers an OpenAI-compatible API, supporting chat completions with reasoning, tool calling, and streaming responses. Developers can control reasoning depth per request and integrate the endpoint into various agentic pipelines or fine-tune the model for specific domains. Monitoring metrics such as request latency and GPU utilization helps optimize performance.

This deployment demonstrates a scalable approach to hosting trillion-parameter models on AWS infrastructure, enabling advanced AI capabilities for research and enterprise applications.