How Do You Know Your Agent Is Good -- Before It Reaches Users?

Part 2 of 6 — “Governing AI Agents in the Enterprise: A Practical Architecture Guide”

This article covers the Evaluationdiscipline: how to build a system that catches quality regressions in an AI agent before they reach customers — and how to define “quality” for a system that does not have a single right answer.

2.1 - Every time we tweak a prompt or swap a model, something breaks somewhere else. How do we catch agent regressions before they reach customers?

The pattern: Evaluation-as-a-CI-gate.

This is the single most important practice in agent operations. The core idea is to treat agent quality the way you treat code correctness: you build a versioned suite of test cases, attach measurable thresholds to them, and run them automatically on every change. A drop below threshold fails the build and blocks the merge — exactly like a failing unit test.

You curate a golden dataset— a set of representative inputs paired with reference answers or expected behaviors. You define metrics with explicit pass thresholds (for example: task completion ≥ 0.75, tool correctness ≥ 0.9, hallucination ≤ 0.3). A tool like DeepEval runs these tests in your CI/CD pipeline on every pull request and every merge to main. Any metric falling below its threshold fails the build.

The same suite runs whenever any input changes: a prompt edit, a model upgrade, a new tool, a dependency bump. That is what catches the “something breaks somewhere else” problem — the suite is the safety net under a non-deterministic system whose source code is its prompts and models.

How to build your first golden set. Pull 20 real inputs from your production logs or hand-craft representative ones, write down what a correct outcome looks like for each — expected tools, expected response shape, hard constraints — and you have a usable starting point. The set grows continuously from there.

import pytest
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric, ToolCorrectnessMetric
from deepeval.test_case import ToolCall
from my_agent import run_agent   # your ADK agent entrypoint

dataset = EvaluationDataset(goldens=[
    Golden(
        input="Book the cheapest NYC->LA flight tomorrow",
        expected_tools=[
            ToolCall(name="search_flights"),
            ToolCall(name="book_flight"),
        ],
    ),
])

task_completion  = TaskCompletionMetric(threshold=0.75, model="gpt-4o")
tool_correctness = ToolCorrectnessMetric(threshold=0.9)

@pytest.mark.parametrize("golden", dataset.goldens)
def test_agent(golden: Golden):
    run_agent(golden.input)        # traced via @observe decorators
    assert_test(golden=golden, metrics=[task_completion, tool_correctness])

name: agent-eval-gate
on: [pull_request]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install deepeval
      - env: { OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} }
        run: deepeval test run test_agent.py   # non-zero exit blocks the merge

Real-world examples

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

policy_adherence = GEval(
    name="No eligibility promise",
    criteria=(
        "The response must NOT state or imply that the applicant is "
        "approved, pre-approved, or eligible for the loan."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.9,
)

2.2 - Our agent's behavior is a multi-step conversation, not a single answer. How do we even define 'correct' and test it?

The pattern: Agentic evaluation — score the trajectory, not just the output.

A retrieval chatbot has one input and one output, so “correct” is straightforward to define. An agent has a trajectory: it plans, calls tools in sequence, observes results, re-plans, and may carry on a multi-turn conversation. “Correct” must therefore be measured along several axes:

• Task completion— did the agent actually achieve the user’s goal across the whole interaction?
• Tool correctness— did it call the right tools?
• Argument correctness— did it pass the right parameters to those tools?
• Plan adherence and plan quality — was its plan sound, and did it stick to it?
• Task efficiency— did it reach the goal without unnecessary steps, which cost money and latency?

For multi-turn behavior, a framework like DeepEval can simulateconversations rather than relying only on hand-written transcripts, and it can synthesize large golden datasets from a small seed set — giving you breadth of coverage that manual authoring cannot match.

from deepeval.conversation_simulator import ConversationSimulator
from deepeval.metrics import TurnRelevancyMetric
from my_agent import model_callback

simulator = ConversationSimulator(model_callback=model_callback)
test_cases = simulator.simulate(goldens=dataset.goldens, max_turns=10)
# each simulated multi-turn case is scored like any other test case in CI

Real-world examples

What to carry forward

Two principles from this article shape everything that follows:

• The golden dataset is a living artifact. It must grow continuously — ideally fed by surprising or problematic production traces. That is why the telemetry layer in the next article matters so much.

• The CI gate is the line between “we tested it” and “we can prove it cannot regress.” Without the gate, every deployment is a gamble. With it, quality degradation cannot quietly reach customers.

Evaluation tells you whether the agent is good. The next article addresses how to see what it is actually doing once it is live — because passing all your tests is not the same as behaving well with real users.