Choose Evaluation Checks From the Agent's Job
Shashank Agarwal
A metric is useful when it answers a question about the agent's job. Starting with a large collection of checks can produce a busy report without establishing whether the user received the right result.
Quick answer: Write the responsibilities of the workflow, identify the evidence available for each responsibility, then choose deterministic checks, model grading or human review accordingly. Mark requirements that cannot yet be observed. Add instrumentation or a controlled fixture before treating those requirements as tested.
Map responsibilities to evidence
For an illustrative inventory assistant, the responsibilities might be to find the requested item, use the correct location and explain whether a reservation actually occurred.
| Responsibility | Evidence to provide | Candidate check |
|---|---|---|
| Use the requested location | Request and tool arguments | Exact field comparison |
| Reserve available stock | Controlled inventory and operation result | State assertion |
| Explain the observed result | Final answer and operation evidence | Focused rubric |
| Clarify an ambiguous item | Conversation and candidate list | Scenario review |
These are proposed checks, not a report of a deployed integration.
Do not ask a grader to invent missing state
A model can inspect a message saying a reservation succeeded. That does not establish that inventory changed. If the tool returns only a generic acknowledgment, define what that acknowledgment guarantees or capture additional evidence.
An output schema can be checked mechanically. Whether an explanation makes an appropriate commitment may require judgment. Keeping those checks separate makes a failed result easier to act on.
Test the check itself
Give a proposed check an intentional passing example and a failing example. Include a near miss: the right item reserved at the wrong location, for instance. Inspect whether the evaluator distinguishes them for the reason you intended.
Record the check's version and the fields it consumed. A revised tool response can otherwise change a result without an agent change.
Keep scope visible
The ADK criteria documentation separates different kinds of trajectory and response evaluation. Read the configured criterion's actual semantics before treating its name as a guarantee.
I'm Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. Start with the first-dataset guide, then use the tool-boundary guide to identify missing evidence.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.