Write the Acceptance Criterion Before the Prompt
Shashank Agarwal
A prompt describes the behavior you want. An acceptance criterion makes the result reviewable. If the criterion is written only after seeing a promising demo, it is easy to reward the answer the system happened to produce.
Quick answer: Define the requested outcome, the evidence that establishes it and the behavior required when the outcome cannot be reached. Keep action permission, factual support and presentation quality as separate conditions. Then write a prompt and test whether the captured result meets those conditions.
Turn a broad instruction into a state change
Consider an illustrative address-change assistant. “Help the customer update their address” leaves several questions unanswered. Does showing a proposed address count? Must the system of record change? What if the customer has not confirmed the new value?
A more reviewable contract is:
| Condition | Evidence a reviewer should inspect |
|---|---|
| The intended account is identified | Approved account context and relevant identifier |
| The new value is confirmed | The actual confirmation of the displayed proposal |
| The change succeeds | The recorded operation and resulting address state |
| Permission is absent | No update occurs; the assistant explains what is needed |
| The update fails | The assistant reports the failure without claiming completion |
This is an invented engineering example, not a description of a Noveum customer or a live integration.
Choose evidence before choosing a score
The final sentence “Your address is updated” does not prove that the update occurred. A tool returning successfully may also be insufficient if the application accepted the request but did not apply it.
Decide which observable state establishes completion. Save an appropriately minimized reference to that state with the relevant operation. A presentation check can assess the explanation, but it should not cancel a failed action condition.
Microsoft’s evaluation checklist includes acceptance criteria and test methods in an evaluation case. The specific state-and-permission table above is my proposed way to make that requirement concrete for an action workflow.
Include the unsuccessful path in the contract
An acceptable result can be a completed change, a request for missing information or a clear failure report. Which one is correct depends on the input and permission state. Write those branches before judging the agent.
Keep a case where the customer declines the proposed address. Keep another where the backend rejects it. Otherwise a system that confidently claims success can look better than one that accurately reports a limitation.
Use the contract at release review
Compare candidates on the same reviewed cases and record the application version and configuration. Preserve individual required-condition results. A better tone score does not establish that an unauthorized change became acceptable.
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. This is the kind of evidence definition I recommend before configuring evaluations. The eval-driven development guide explains the surrounding loop; the HTTP-success example shows why an operation’s response and a task’s outcome can disagree.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.