Separate Agent Format Errors From Task Failure
Shashank Agarwal
Valid JSON can contain the wrong answer. A correct answer can also arrive in a form that a downstream system cannot consume. Evaluation should preserve both facts.
Quick answer: Test parseability and schema compliance separately from semantic correctness, action outcome and downstream acceptance. Keep the raw output and any transformed output. If a repair step changes the response, evaluate the repaired artifact too and record which component produced it.
Use a concrete contract
Consider an illustrative routing service that expects an object containing a destination and a reason. A response can satisfy the schema while choosing the wrong destination. Another response can name the right destination in prose but violate the machine interface.
Neither result should be reported simply as “the model passed.” The consumer's requirements determine whether the overall task succeeded.
Keep the result dimensions distinct
| Check | Question answered |
|---|---|
| Parsing | Can the output be read as the required serialization? |
| Schema | Are required fields and types present? |
| Semantics | Are the values correct for the supplied task? |
| Execution | Did an intended operation occur as expected? |
| Consumer acceptance | Did the next system accept the result? |
Use deterministic checks for conditions that are fully specified. Apply focused review where the task requires judgment. Do not ask a language-model grader to substitute for an available parser or state assertion.
Do not hide automatic repair
A wrapper may strip extra text or request a corrected response. Preserve the original failure and the final artifact, including extra attempts where observable.
If the wrapper invents a missing destination, it has made a semantic decision. That decision needs evaluation; a clean final schema does not establish its correctness.
Test the boundary deliberately
Include a correct valid object, a valid object with an incorrect value, an invalid structure with a recoverable syntax issue and an incomplete result. Decide how the real consumer should treat each before grading.
These are proposed fixtures, not production measurements or a claim about any specific provider's structured-output guarantees.
I'm Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. The acceptance-criteria guide helps define the contract; the tool-boundary guide helps preserve what the consumer actually received.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.