Building Reliable AI Agents with Evaluation Pipelines on AWS

Motorway, a UK-based online car marketplace, collaborated with AWS to create an end-to-end evaluation framework for their AI dealer search agent. This agent allows dealers to find vehicles using natural language queries, replacing manual filtering processes. The challenge was ensuring the agent's responses are accurate and trustworthy, especially when incorrect tool use or semantic misinterpretations could lead to wrong results.

The solution involved building a pipeline that combines the open-source Strands Agents SDK with Amazon Bedrock's fully managed AgentCore service. This setup enables rigorous pre-deployment testing across three evaluation layers—tool usage, reasoning, and output quality—each with specific performance thresholds. The pipeline also incorporates production monitoring, using Amazon CloudWatch and AgentCore Evaluations, to continuously assess agent performance during live operation.

This evaluation framework has led to substantial improvements: tool selection accuracy increased from 87% to 98%, and the number of monthly incidents dropped from 12 to 2. The dealer search process now takes minutes instead of hours, with higher confidence in the results.

The approach emphasizes the importance of multi-layer testing, multi-trial evaluation to address non-determinism, and ongoing production monitoring. It also advocates for turning real user interactions into test cases, creating a feedback loop that enhances agent reliability.

This methodology is applicable across various customer-facing AI systems, including service bots, financial advisors, and healthcare assistants, helping developers deploy more dependable AI solutions at scale.