Automated Evaluation of AI Agents Using Amazon Bedrock AgentCore and GitHub Actions

Amazon Bedrock AgentCore offers a managed platform for deploying and evaluating AI agents. This article explains how to build a CI/CD quality gate that deploys an AI agent and an OAuth-protected MCP server to the AgentCore runtime, invokes the agent with test prompts, scores the responses, and automatically blocks pull requests when agent behavior regresses.

The pipeline involves deploying the agent and MCP server using AWS CDK, authenticating via a shared Cognito user pool, and invoking the agent with evaluation prompts through a unified script. The evaluation uses Amazon Bedrock AgentCore's Evaluate API, which scores agent responses based on dimensions like helpfulness and correctness, using large language models as judges.

A key challenge addressed is authenticating CI pipelines against OAuth-protected MCP servers without user context. The solution supports three approaches, with this implementation using M2M tokens that bypass role checks, suitable for automated testing. The pipeline automatically assesses whether the agent's performance meets predefined thresholds, blocking pull requests that regress.

The setup includes deploying the necessary infrastructure via CDK, configuring role-based access control, and using a unified Python script for evaluation. The GitHub Actions workflow automates deployment, evaluation, and cleanup, ensuring that only agents passing quality gates are merged.

This approach provides a scalable, automated method for maintaining AI agent quality, reducing manual testing effort, and catching regressions early in the development process. The accompanying repository offers complete code and deployment scripts for replication and extension.