models
Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
Amazon SageMaker Inference introduces prefix-aware routing, a new strategy that enhances large language model (LLM) request handling by directing requests with similar prefixes to the same instance, significantly reducing latency and increasing cache hit rates.
AS1 NewsSource: aws.amazon.com
AMZNAmazon SageMaker Inference has launched a new routing strategy called prefix-aware routing, designed to optimize the performance of large language model (LLM) inference workloads. This strategy ensures that requests sharing the same prompt prefix are consistently routed to the same instance, allowing the key-value (KV) cache to stay warm and be reused effectively.
In typical LLM serving scenarios, requests often contain a fixed prompt setup followed by variable user input. When scaled across multiple instances, this setup can lead to inefficient cache utilization, as each instance recomputes the shared prefix independently. Prefix-aware routing addresses this by analyzing the beginning of each request and directing similar requests to the same instance, thus maintaining a warm cache and reducing redundant computation.
Benchmarks on Llama 3.1 70B models demonstrate the effectiveness of this approach. The new routing strategy reduced the 50th percentile time-to-first-token (TTFT) by up to 77% and increased KV cache hit rates from approximately 25% to over 80%. Additionally, throughput improved by up to 16%, with minimal routing overhead of about 1.3 to 1.9 milliseconds per request.
The feature is configurable during endpoint creation, allowing users to specify the length of the prefix used for routing and the maximum number of in-flight requests per instance. It seamlessly integrates with existing SageMaker endpoints and does not require changes to the model container or serving framework.
Prefix-aware routing is particularly beneficial for applications involving retrieval-augmented generation, multi-turn conversations, templated bots, and code completion, where shared prefixes are common. It supports multi-tenant scenarios by allowing requests with different tenant IDs to be routed independently.
Overall, this new routing strategy enhances the efficiency and scalability of LLM inference on SageMaker, enabling faster response times and better resource utilization for demanding AI applications.
Improves inference latency and cache efficiency for large language models by routing similar requests to the same instance, reducing redundant computation and enhancing throughput.