Build a Physical AI Model Factory with NVIDIA Cosmos 3 on SageMaker HyperPod

Building a Physical AI system, such as a robot or autonomous vehicle, requires a continuous pipeline rather than a single training job. This pipeline involves synthetic data generation, post-training perception and policy models, and closed-loop evaluation. NVIDIA Cosmos 3, a versatile omnimodal world model, enables this process by unifying generation, post-training, and evaluation stages into a single, persistent GPU node pool.

Cosmos 3 employs a Mixture-of-Transformers (MoT) design with per-layer joint attention and a deliberate asymmetry between training and inference workloads. It handles video, image, action, and sound as a single token stream, allowing seamless integration across modalities. The model runs in three modes—world modeling, inverse dynamics, and policy deployment—making it suitable for end-to-end Physical AI model factories.

The architecture's key features include a shared token stream, dual experts joined by attention, and asymmetric inference modes. During training, Cosmos 3 predicts future video frames and actions, while during deployment, it runs fewer denoising steps and skips pixel decoding for efficiency. This design simplifies workload management and resource sharing.

Implementing this system on Amazon SageMaker HyperPod with Amazon EKS provides a unified, resilient cluster environment. It supports running multiple stages—generation, post-training, and evaluation—on a shared GPU pool with a common storage layer, reducing data movement and provisioning overhead. HyperPod's auto-recovery and capacity management features ensure high availability and cost efficiency.

The article details the setup process, including prerequisites, cluster configuration, and workload orchestration. It demonstrates running three representative workloads—robot policy post-training, vision perception fine-tuning, and low-rank adaptation—on a single cluster, validating the approach with real checkpoints and datasets.

This approach enables continuous operation of the Physical AI flywheel, optimizing GPU goodput and reducing operational complexity. The provided repository offers infrastructure templates and job manifests to replicate the setup, facilitating adaptation to various robotic or autonomous vehicle applications.

For further information, readers are encouraged to explore NVIDIA Cosmos, the cosmos-framework repository, and Amazon SageMaker HyperPod documentation, with sample datasets and code available on GitHub.