Reduce RAG costs on Amazon Bedrock with query-aware compression

Retrieval Augmented Generation (RAG) workflows often incur high costs due to the large number of input tokens processed by foundation models. Amazon Bedrock introduces a query-aware context compression pattern that significantly reduces these costs. After retrieval, a smaller, less expensive model filters the retrieved chunks against the user's query, passing only the relevant spans to the primary model for answer generation.

This approach involves a single AWS Lambda function orchestrating two Amazon Bedrock model calls: first to the smaller compression model, then to the primary model. The compression model extracts only the evidence relevant to the query, which reduces the number of tokens the primary model needs to process. Empirical evaluations show that this method can achieve up to 36% cost savings and a 10-fold reduction in context tokens, with minimal impact on answer correctness and a notable decrease in hallucination rates.

Implementing this pattern requires setting up IAM roles, ensuring access to the appropriate models, and designing prompts that instruct the smaller model to extract verbatim evidence. The pattern is particularly effective when the retrieved context is large, the primary model is costly, and questions are narrow, such as in regulated industries, customer support, internal documentation, or financial analysis.

While adding an extra model call introduces some latency, the speed of the smaller model and the focused context can mitigate this. Proper evaluation with representative queries is recommended before deployment. This pattern complements existing Bedrock features like prompt caching and reranking, enabling comprehensive optimization of RAG workflows.