models
Configure Rate Limits for AI Traffic on Amazon Bedrock AgentCore Gateway
Amazon Bedrock's AgentCore gateway now supports fine-grained rate limiting controls, enabling organizations to manage AI traffic effectively. This feature allows setting request, token, and connection limits scoped by user identity or target, helping to protect downstream models and services from traffic spikes.
AS1 NewsSource: aws.amazon.com
AMZNAmazon Bedrock's AgentCore gateway is a fully managed, serverless AI gateway that provides a secure entry point for AI traffic, routing requests to various tools and models. The recent addition of support for rate limiting enhances its capability to control traffic flow, ensuring service availability and fairness.
Organizations can now define OAuth or IAM-based rules to limit requests per minute, concurrent connections, and token throughput, scoped by JWT claims or IAM identity. This granular control helps prevent overloads on downstream models, tools, and agents, especially during traffic surges.
The gateway supports three target types: MCP targets, inference targets, and HTTP passthrough targets. Rate limiting metrics include request rate limits (requests per second or minute), token rate limits (tokens per minute, applicable to inference targets), and connection rate limits (connections per second). These limits are enforced using dimension keys such as targetName, toolName, qualifiedModelId, and JWT claims.
Configuring rate limits involves creating rules with specific dimension keys and entries, which define the allowed throughput for each bucket. For example, limits can be set per user role, individual user, specific target, or model, enabling layered and highly granular traffic management.
The system evaluates customer-defined rate limits before applying service quotas, with the effective rate being the minimum of the two. It also supports multi-dimensional limits, combining user identity and target to enforce precise control. These configurations are managed via CLI commands, allowing for flexible and programmatic setup.
Monitoring and debugging are facilitated through OpenTelemetry span attributes, providing insights into rate limit evaluations and denials. This observability supports operational confidence and helps optimize traffic policies.
Overall, the new rate limiting features in Amazon Bedrock's AgentCore gateway enable organizations to operate AI services at scale with improved reliability, fairness, and protection against traffic spikes.
Enhances traffic management and service reliability for AI workloads by enabling detailed rate limiting controls.