models
Benchmarking Small LLM Inference on SageMaker AI: G7 vs G5 and G6
This article compares the performance of different GPU instances—G5, G6, G6e, and G7—on Amazon SageMaker AI for deploying 30B Mixture-of-Experts models, highlighting how G7's NVIDIA Blackwell GPUs offer measurable improvements in throughput, latency, and cost efficiency.
AS1 NewsSource: aws.amazon.com
COST
NVDA
AMZNChoosing the right GPU instance is crucial for deploying large language models (LLMs) efficiently at scale. Recent benchmarking on Amazon SageMaker AI demonstrates that the latest G7 instances, powered by NVIDIA Blackwell GPUs, deliver significant performance gains over previous generations.
In the first use case, the Qwen3-Coder-30B model was deployed across G5, G6, and G7 instances to evaluate throughput, latency, and cost. The G7 instance, with two GPUs and 64 GB of memory, achieved approximately 391.3 output tokens per second, outperforming G5 and G6 configurations by 13% and 60.8%, respectively. It also reduced average request latency by about 37.6% compared to G6.
The second use case involved deploying NVIDIA Nemotron-3-Nano-30B for reasoning and question-answering tasks, using Amazon SageMaker AI Generative AI Inference Recommendations. This automated process identified G7 configurations as the most cost-effective and high-performing options, with the ml.g7.48xlarge providing the highest throughput and ml.g7.2xlarge offering the lowest estimated cost per million output tokens.
These benchmarks underscore the importance of workload-specific testing when selecting inference infrastructure. The G7 instances, with native FP4 support and higher memory bandwidth, are particularly advantageous for memory-bound MoE architectures, enabling lower latency and higher throughput.
Overall, the results confirm that G7 instances can deliver superior price-performance for real-time LLM inference, making them a compelling choice for enterprise AI deployment. Users are encouraged to benchmark their specific workloads to determine the optimal configuration, leveraging tools like Amazon SageMaker AI Generative AI Inference Recommendations.
Cleanup procedures involve deleting the created endpoints and models to avoid ongoing charges, which can be managed via the AWS console or SDKs. The benchmarks and configurations discussed are available in accompanying notebooks for reproduction and further analysis.
The benchmarks demonstrate that G7 GPU instances provide measurable improvements in large language model inference performance, latency, and cost efficiency, influencing deployment decisions for enterprise AI applications.