How Do You Know Your Agent Is Good -- Before It Reaches Users?
Every time we tweak a prompt or swap a model, something breaks somewhere else. How do we catch regressions before customers see them?
5 min · Updated June 2026
Article 2 of 6 — “Governing AI Agents in the Enterprise: A Practical Architecture Guide”
This article covers the Evaluationdiscipline: how to build a system that catches quality regressions in an AI agent before they reach customers — and how to define “quality” for a system that does not have a single right answer.
Q2.1 -- Every time we tweak a prompt or swap a model, something breaks somewhere else. How do we catch agent regressions before they reach customers?
The pattern: Evaluation-as-a-CI-gate.
This is the single most important practice in agent operations. The core idea is to treat agent quality the way you treat code correctness: you build a versioned suite of test cases, attach measurable thresholds to them, and run them automatically on every change. A drop below threshold fails the build and blocks the merge — exactly like a failing unit test.
You curate a golden dataset— a set of representative inputs paired with reference answers or expected behaviors. You define metrics with explicit pass thresholds (for example: task completion ≥ 0.75, tool correctness ≥ 0.9, hallucination ≤ 0.3). A tool like DeepEval runs these tests in your CI/CD pipeline on every pull request and every merge to main. Any metric falling below its threshold fails the build.
The same suite runs whenever any input changes: a prompt edit, a model upgrade, a new tool, a dependency bump. That is what catches the “something breaks somewhere else” problem — the suite is the safety net under a non-deterministic system whose source code is its prompts and models.
Real-world examples
Banking -- loan pre-qualification agent
A prompt change intended to make the agent friendlier inadvertently makes it more willing to give applicants optimistic eligibility guidance. A golden dataset of edge-case applicant profiles, each with a correct eligibility outcome, catches the regression in CI before a single customer sees it. The metric that fails: a custom policy-adherence check that the agent never states or implies an approval decision.
Healthcare -- clinical intake and triage agent
The hospital upgrades to a newer foundation model expecting improvement across the board. On the regression suite, overall quality rises — but the tool-correctness metric drops: the new model occasionally skips calling the symptom-severity scoring tool and reasons unaided instead. The CI gate blocks the upgrade until the prompt is adjusted. The patient-safety incident never happens.
Retail -- product recommendation agent
A merchandising team edits the agent prompt to promote a seasonal category. The regression suite shows recommendation relevance holding steady, but a “no out-of-stock items” check fails — the new instruction nudged the agent away from checking inventory. Caught in the pull request, fixed before deployment.
Q2.2 -- Our agent's behavior is a multi-step conversation, not a single answer. How do we even define 'correct' and test it?
The pattern: Agentic evaluation — score the trajectory, not just the output.
A retrieval chatbot has one input and one output, so “correct” is straightforward to define. An agent has a trajectory: it plans, calls tools in sequence, observes results, re-plans, and may carry on a multi-turn conversation. “Correct” must therefore be measured along several axes:
- Task completion— did the agent actually achieve the user’s goal across the whole interaction?
- Tool correctness— did it call the right tools?
- Argument correctness— did it pass the right parameters to those tools?
- Plan adherence and plan quality — was its plan sound, and did it stick to it?
- Task efficiency— did it reach the goal without unnecessary steps, which cost money and latency?
For multi-turn behavior, a framework like DeepEval can simulateconversations rather than relying only on hand-written transcripts, and it can synthesize large golden datasets from a small seed set — giving you breadth of coverage that manual authoring cannot match.
Real-world examples
Travel and hospitality -- trip-booking agent
“Book me a flight and hotel under budget, near the conference venue.” Success is not one answer; it is a sequence: search flights, check the budget, search hotels filtered by location, confirm. Task-completion and tool-correctness metrics verify the whole trajectory; the task-efficiency metric flags a version of the agent that searched hotels five times because it kept forgetting the budget constraint.
Human resources -- employee onboarding agent
A new hire’s onboarding spans many turns and systems: provision accounts, enroll in benefits, schedule training, assign a buddy. A simulated multi-turn evaluation checks that the agent completes every required step and never skips the benefits-enrollment deadline. Plan-adherence catches an agent that re-orders steps in a way that breaks a dependency.
Telecom -- technical-support agent
Diagnosing a connectivity fault is inherently multi-step: check line status, run a remote diagnostic, interpret results, decide between a config push and a truck roll. Argument-correctness verifies the agent passes the correct account and circuit identifiers to the diagnostic tool — a wrong argument here means diagnosing the wrong customer’s line.
What to carry forward
Two principles from this article shape everything that follows:
- The golden dataset is a living artifact. It must grow continuously — ideally fed by surprising or problematic production traces. That is why the telemetry layer in the next article matters so much.
- The CI gate is the line between “we tested it” and “we can prove it cannot regress.” Without the gate, every deployment is a gamble. With it, quality degradation cannot quietly reach customers.
Evaluation tells you whether the agent is good. The next article addresses how to see what it is actually doing once it is live — because passing all your tests is not the same as behaving well with real users.