Keep an Agent Evaluation Rubric Understandable
Shashank Agarwal
A rubric becomes difficult to use when it asks reviewers to judge several unstated qualities at once. “Helpful, accurate and appropriate” is a direction, but it is not yet a repeatable decision rule.
Quick answer: Give each criterion one observable responsibility, name the evidence it requires and describe passing, failing and insufficient-evidence outcomes. Include a boundary example. Version the rubric and review disagreements before adding more criteria or changing the threshold.
Replace adjectives with observable behavior
For an illustrative service-status assistant, replace “gives a good answer” with “reports the status shown by the supplied lookup and identifies any lookup limitation.”
This leaves room for different wording while preserving the task requirement. Do not require an exact phrase unless the wording itself is part of the contract.
Make the criterion inspectable
| Field | Example |
|---|---|
| Responsibility | Describe the observed service state accurately |
| Required evidence | User request, lookup result and final response |
| Pass | Explanation agrees with the supplied result |
| Fail | Explanation contradicts the supplied result |
| Insufficient evidence | Lookup result is unavailable or uninterpretable |
| Boundary | A queued operation is described as queued, not completed |
The example is a proposed rubric, not a measured application result.
Keep criteria from overlapping silently
If accuracy and evidence consistency test the same observation, a combined score may count one issue twice. Overlap can be intentional, but reviewers should know why it exists.
Separate language quality from claims about state. A polished explanation does not compensate for a wrong status.
Review disagreements as design feedback
Give reviewers the same packet and compare their reasons. If they disagree about whether a queued operation counts as success, clarify the task contract. If a required field was absent, improve the packet.
Record the change and rerun affected cases. A new rubric version can change the apparent result even when the agent is unchanged.
I'm Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. I recommend keeping a short decision log beside the rubric so later reviewers understand why boundary rules exist.
Use the acceptance-criteria guide to define the task and the first-dataset guide to select examples that exercise its boundaries.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.