The North Star
Every organization racing to deploy AI agents faces the same reckoning: an agent that demos beautifully can quietly degrade in production, and by the time a human notices, the trust is already broken. Where traditional software fails loudly, agents fail softly. They return an answer that looks confident, reads fluently, and is wrong. When that answer drives a real business decision, the cost is not a broken feature. It’s a broken relationship with the user.
That’s the quality problem at its core. The challenge is therefore not building agents that work once, but proving they keep working, at scale, across every sub-agent, every workflow, and every release. The teams that win the agent era will be the ones who treat quality not as a final QA step but as the operating system of the agent itself: continuously measured, automatically defended, and improved with every interaction. In this post, we show how we built a continuous quality loop for production-ready agentic solutions at scale using Amazon Bedrock AgentCore.
Business problem
The solution uses Amazon Bedrock AgentCore and other AWS services for internal use, so sellers and sales leaders can ask business questions in plain English from Amazon Quick. Behind that single experience, several specialized sub-agents work together: the Performance Insights Agent for territory-scoped revenue and performance, the Customer Insights Agent for customer intelligence, the Data Insights Agent for text-to-SQL over foundational datasets, and direct dashboard access. Answers are scoped to each user’s territory and role-level security. Field teams use these answers to assist in planning account strategy, prepare for customer meetings, so the numbers must be right. The most damaging failure is not a crash. It’s a fluent, confident answer that is quietly wrong. Before launch, when we evaluated the Customer Insights Agent, we found exactly that:
- It reported $410.08 of a customer’s three-month revenue on a service as “$410.08 Million.”
- It reported Sales Revenue labeled as Sales Revenue excluding Adjustments.
- It offered an Annual Recurring Revenue estimate that appeared nowhere in the data.
- It presented its own calculations as sourced facts.
Through the different runs, monitoring stayed green. It told us the system had responded, not whether the answer was right.
Had these errors shipped uncaught, the field would have paid for them. A seller who repeats an inflated figure can lose credibility with the customer. This is exactly why we caught it before launch. The risk doesn’t end at launch: it grows with every new agent and every change to data, prompts, code, or models. It also grows without any release, because upstream pipelines refresh the data every week. Coordinated releases introduce a related risk: when a data change, a tool change, and a paired prompt update are meant to ship together but land apart, the system runs mismatched until all changes catch up. In one pre-production test, that mismatch window by itself dropped the pass rate from 78 percent to 38 percent. To catch failures like these, before every release and continuously in production, we need a framework that can show whether answers are grounded and correct, and trace each failure back to its root cause in the data, the prompt, or the agent’s code.
At AWS, the Sales, Marketing, and Global Services (SMGS) Operations team developed a production multi-agent system serving AWS Field Sales worldwide. It serves two distinct audiences. For field sales teams, it replaces hours of manual data analysis with on-demand, conversational intelligence, helping sellers understand the health of their business, surface risks and opportunities, and take action faster. For leadership, it powers business management cadences including Weekly Business Reviews (WBRs), surfacing the signals that matter across the portfolio without requiring manual report preparation. The same agentic system serves both the seller on the front line and the leader reviewing performance at scale.
The agentic system orchestrates a growing portfolio of specialized sub-agents, each owning a distinct capability domain. Among them: the Performance Insights Agent generates natural-language business intelligence narratives from complex sales data, turning raw metrics into actionable seller insights. The Customer Insights Agent assembles a 360-degree view of each customer by synthesizing signals across product usage, support history, contract status, and engagement data, helping sellers understand where a customer is in their journey and what they need next. The Data Insights Agent is a text-to-SQL agent that account teams use to query structured analytics data in plain English without writing a single line of SQL. These three agents are the ones referenced throughout the rest of this post.
Coordinating these agents reliably at scale made one thing clear: agentic quality wasn’t an operational concern to manage later. It was the prerequisite for operating. A single degraded sub-agent can silently corrupt every response downstream, and traditional monitoring won’t catch it. That reality made a full lifecycle evaluation framework not optional but necessary. AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, is the connective tissue powering it, alongside Amazon Bedrock Guardrails (safety), Amazon CloudWatch (metrics, alarms, tracing), Amazon Simple Storage Service (Amazon S3) and Amazon Redshift (evaluation persistence), Amazon Quick (dashboards), and AWS Lambda (custom evaluators).
In this post, we share how we tackled the hardest operational challenges in running AI agents at enterprise scale, and how Amazon Bedrock AgentCore Evaluations became the backbone of agent quality. Specifically, we cover:
- Agents play by different rules – Why agents demand a different operational discipline, and why traditional monitoring (latency, error rates, uptime) leaves you blind to the failures that matter most.
- A map for the whole journey – The nine-stage evaluation framework we built on AgentCore Evaluations, covering every phase of the agent development lifecycle from first commit to production.
- From theory to practice – Concrete patterns (continuous integration and continuous delivery (CI/CD) regression gates, canary scoring, drift detection, and automated feedback loops) and exactly how AgentCore Evaluations powers each one.
- Metrics that actually matter – The measurable impact on deployment confidence, regression prevention, and iteration speed across our agent portfolio.
Why agents need a structured evaluation framework
Traditional software is deterministic: the same input produces the same output. Machine learning (ML) models introduced probabilistic outputs, and MLOps emerged to manage their training, versioning, monitoring, and re-training. AI agents add a third layer of operational complexity: autonomous, multi-step reasoning, where the question shifts from “did something break?” to “is the agent making sound decisions across its entire reasoning chain?”
Agents inherit the hard problems of both predecessors, and introduce some new ones. They’re non-deterministic, following different reasoning paths for identical inputs. They make cascading decisions across routing, tool selection, and synthesis, where a single step can fail quietly and corrupt everything downstream. Most critically, they degrade silently: latency stays low, error rates stay flat, yet the quality of agent response erodes, because the agent is still responding. It is just responding wrong.
Agent evaluation is the practice of catching exactly that. It spans five dimensions: safety (is it behaving within defined boundaries?), quality (is the agent producing correct, coherent, grounded responses?), performance (is it fast and reliable enough?), cost (is it running efficiently at scale?), and operational control (can we detect, diagnose, and remediate issues quickly?). A mature evaluation practice addresses five continuously, throughout the full agent lifecycle.
In this post, we share how we operationalize five evaluation dimensions across a production multi-agent system. Safety and security is our first priority. No agent ships, and no agent keeps running, if it can’t meet its safety bar. Quality gets the most space in this post because it’s the hardest of the five to get right and benefits most from a dedicated framework, but it doesn’t outrank safety. Performance, cost, and operational control are woven into every stage of the lifecycle alongside both. Critically, this framework applies at every level of the agent hierarchy. At the tool level, each capability (a customer disambiguation tool, graph retrieval tool) has its own evaluation contract. At the agent level, each sub-agent is evaluated end-to-end. At the multi-agent level, orchestration paths are evaluated for coherence, correct routing, and synthesis accuracy across agent boundaries.
The nine stages apply at three levels simultaneously, and this hierarchy is not specific to this product. It’s the right mental model when we build or integrate new production agents. Most teams start by evaluating at the agent level and stop there. But agent-level quality is only as good as the tools it relies on, and multi-agent systems introduce a third failure surface that neither level can see alone. A robust evaluation practice covers three, with each level running its own golden set, its own evaluators, and its own promotion criteria through the nine stages.
The agent evaluation framework: nine stages, one closed loop
These nine stages trace the agent lifecycle from first commit to live production, and each one answers a question teams face as their agents mature.
The agent lifecycle and the nine common questions
Every team building production agents encounters the following questions as their agent matures:
Stage 1: The Foundation: “What does good look like?”
Stage 2: The Experiment: “Which version is better?”
Stage 3: The Merge (CI): “Did I break something?”
Stage 4: The Gate (CD): “Is it safe for production?”
Stage 5: The Canary: “Did something silently break?”
Stage 6: The Adversary: “What if someone tries to break it?”
Stage 7: The Drift: “Is quality drifting in live?”
Stage 8: The Pulse: “How do we know it’s working right now?”
Stage 9: The Feedback Loop: “How does it get better automatically?”
We build one framework for every stage of the agent lifecycle using AgentCore Evaluations. The nine stages are not separate tools stitched together after the fact. They’re nine points on a single, continuous evaluation surface, powered by AgentCore Evaluations from first commit to production response, as shown in Figure 1.
Figure 1: The nine-stage AgentCore Evaluations journey, from Define through Improve
Here’s how we tackle each one: the pattern that works, the AgentCore services behind it, and what it looks like running in production, shown in system design in Figure 2.
Figure 2: System design of the AgentCore evaluation framework
Stage 1: The Foundation. Define quality
The foundation of evaluating an agent isn’t about choosing the right tools. It’s about asking the right question: what does success look like for this agent, and how do we know when it is failing? The answer must come from the customer, not from the model. This is an iterative process. Our definition of success has sharpened as we observe what the agent actually does in production. Everything defined here (failure thresholds, evaluator stack, golden set, safety rules, latency budgets, and cost targets) is the contract every subsequent stage enforces. If Stage 1 is vague, every downstream gate is guessing.
Stage 1.1: Define your business metrics first. Start by asking what the agent exists to deliver. The two end user personas define success differently:
- For sellers, success means walking into a customer conversation with accurate, current intelligence at the right detail level and at the right time.
- For leadership, success means reliable weekly and cross-business review inputs without lengthy manual preparation. These are business outcomes, not model properties.
Stage 1.2: Translate those business metrics into evaluation metrics. For each outcome, identify the signal that confirms the agent delivered it. Each signal becomes a tracked metric with an explicit threshold and an alarm condition. Accurate account intelligence maps to entity resolution accuracy and grounding rate. Seller trust maps to faithfulness and hallucination rate. Seller action maps to helpfulness and scope adherence. Leadership review reliability maps to accuracy pass rate and response coverage. This translation must cover five dimensions: alongside quality thresholds, define safety guardrail policies, set p50, p90, and p99 latency targets per tool and per agent, establish cost-per-query budgets, and define the service level agreements (SLAs) that determine when an alarm fires and which runbook it triggers.
For accuracy specifically, because the solution directly informs revenue decisions, renewal strategies, and executive reviews, the bar is high. A confidently wrong answer does not only waste a seller’s time. It leads to a wrong decision in a real business moment. We define six failure types derived from real production errors, any one of which marks a response as inaccurate:
- Metric swap: correct customer and period, wrong metric returned. Tracked as occurrences per 100 responses.
- Wrong period: data from one time window is labeled as another, for example a year-over-year (YoY) figure presented to the seller as month-over-month (MoM), or the reverse. Tracked as % citing the correct window.
- Wrong entity: conflated account names or subsidiaries. Tracked as entity resolution accuracy %.
- Pure hallucination: ungrounded figures with no traceable source. Tracked as grounding rate %.
- Wrong symbols: correct metric, wrong currency or unit symbol. Tracked as scope adherence rate %.
- Unauthorized calculation: unsanctioned derived figures not present in the source data. Tracked as occurrences per 100 responses.
The overall accuracy pass rate is the headline metric. Per-dimension tracking is what makes it actionable. Beyond accuracy, we also track Response relevance, Faithfulness, Helpfulness, Instruction Following,Tool Selection Accuracy. A correct answer buried in three paragraphs, or arriving too late, has also failed the seller.
Stage 1.3: Build an evaluator stack that reflects those metrics. No single evaluator type covers the various failure modes possible. We combine four types:
- AgentCore built-in evaluators for broad baseline coverage. Relevance, faithfulness, and helpfulness map directly to evaluation templates available natively in AgentCore Evaluations.
- Customized evaluators, one per failure type in our taxonomy: a metric swap evaluator, a wrong period evaluator, a wrong customer evaluator, a pure hallucination evaluator, a wrong symbols evaluator, and an unauthorized calculation evaluator. None of the six are covered by native templates. They’re domain-specific signals we built on top of the AgentCore framework. Knowing upfront which metrics a native template already covers, and which require a customized evaluator, is itself part of the design decision.
- Third-party evaluators, we deployed the managed evaluators from third-party open source libraries DeepEval and AutoEval.
- Deterministic evaluators for ground truth checks where the correct answer is known exactly, such as revenue, renewal date, and contract status.
Where reasoning quality and helpfulness have no single correct answer, we use LLM-as-judge rubrics instead, with built-in rubrics for relevance, faithfulness, and helpfulness plus support for fully custom rubrics. AgentCore Evaluations provides both ground truth and LLM-as-judge evaluation natively, which is what let us stand up both evaluation paths without building either scoring engine ourselves. Deterministic gives precision on facts. LLM-as-judge gives breadth on reasoning.
Stage 1.4: Build a golden set to exercise your evaluation metrics.