AWS Introduces Query-Aware Compression to Reduce RAG Costs

AWS ML Blog · rss · 2026-08-22

Input tokens often represent a significant cost factor for Retrieval Augmented Generation (RAG) at scale. This AWS ML Blog post introduces a "query-aware compression" pattern on Amazon Bedrock that reduces these costs by filtering retrieved chunks before they reach the primary model.

Architecture Overview:

Retrieval: The retriever searches vectorized sources and returns top-k chunks.
Compression: An AWS Lambda function sends the query and chunks to a smaller, lower-cost model (e.g., Claude Haiku). This model outputs only the verbatim spans directly relevant to the query, discarding irrelevant text.
Generation: The Lambda function sends the query and the "compressed context" to the primary model (e.g., Claude Sonnet) to generate the final answer.

Economic Logic:

Savings are achieved because the expensive primary model processes significantly fewer input tokens ($R/c$ instead of $R$). The approach is cost-effective when retrieved context is large, the price ratio between the large and small models is high, and irrelevant content can be trimmed without affecting answer quality. The article provides a detailed cost formula comparing baseline vs. compressed workflows.

Implementation Details:

The solution is implemented as a single AWS Lambda function orchestrating two Bedrock calls via the Converse API.
A critical COMPRESSIONSYSTEMPROMPT is provided, instructing the small model to extract verbatim sentences/paragraphs only and to preserve chunk identifiers for citation.
The pattern is compatible with Amazon Bedrock Knowledge Bases and can be layered with existing optimizations like Prompt Caching and the Rerank API.

Original post →

AWS Introduces Query-Aware Compression to Reduce RAG Costs

More from Infra