Enhanced KV Cache Architecture for Large Language Models on Amazon SageMaker HyperPod with Curvine

Running large language model (LLM) inference at scale often involves a trade-off between GPU memory capacity and latency. To address this, AWS has developed a tiered KV cache architecture on Amazon SageMaker HyperPod that extends cache sharing beyond GPU memory into a distributed NVMe pool using Curvine. This approach allows multiple model replicas to reuse cached attention keys and values at near-local-disk speeds, significantly improving performance and reducing costs.

The architecture comprises three cache tiers: L0 on GPU memory, L1 on local CPU memory, and L2 on a shared Curvine distributed filesystem pooled from node-local NVMe drives. The L0 cache is the native prefix cache in vLLM, which holds the most frequently accessed KV blocks. When these are evicted, LMCache offloads them to the host DRAM (L1), managed automatically by the SageMaker HyperPod Inference Operator. The L2 tier, powered by Curvine, pools NVMe storage across nodes into a shared namespace accessible by all inference pods, enabling cross-replica cache reuse.

A built-in intelligent router within the HyperPod Inference Operator directs requests to the most appropriate replica based on prefix awareness or cache state, maximizing cache hit rates. The system is configured via the InferenceEndpointConfig CRD, which specifies cache enablement and routing strategies. The setup involves enabling HyperPod Tiered Storage, deploying Curvine on node-local NVMe, and patching the Inference Operator to connect to the Curvine filesystem.

In testing, this architecture achieved up to 100% cross-Pod cache hit rate, a 2.7x improvement in time-to-first-token, and cross-node L2 read latency of approximately 56 ms for a 1,900-token prompt. These results demonstrate that large models previously requiring expensive P5 instances can run efficiently on G6e instances, lowering overall deployment costs.

The solution is suitable for workloads with high prompt overlap, such as retrieval-augmented generation pipelines, multi-turn dialogues, and multi-tenant deployments sharing system prompts. AWS provides detailed guidance on setting up the environment, including enabling HyperPod Tiered Storage, installing the Inference Operator, deploying Curvine, and configuring the cache topology.

This architecture exemplifies how combining advanced caching strategies with shared distributed storage can optimize large language model inference at scale, balancing performance and cost.