Turn an Agent Failure Into a Repeatable Regression Test
Shashank Agarwal
A production failure is often a story: a user asked for something, the agent did something else, and a screenshot shows the result. A regression case needs a more precise record of the conditions that made the failure possible.
Quick answer: Preserve the task, relevant starting state, captured operations and observed mismatch. Replace changing external dependencies with reviewed fixtures where appropriate. Define what should happen under those conditions, then test the failure and nearby cases without silently changing the evidence.
Find the smallest state that explains the mismatch
Consider an illustrative calendar assistant asked to cancel a particular appointment. It finds two similarly named events and selects the wrong event identifier. The final message sounds reasonable, but the selected action does not match the request.
A useful regression fixture retains the requested appointment, the candidate events, the selection criteria and the proposed event identifier. Use synthetic identities and events. Do not perform a real cancellation to create the teaching case.
The failure condition is specific: the proposed identifier points to the wrong event. A broad label such as “calendar reasoning failed” loses the detail a developer needs to test a correction.
Freeze what is supposed to stay fixed
| Record | Why it matters |
|---|---|
| User task | Defines the requested appointment |
| Candidate-event fixture | Prevents the search result from changing between comparisons |
| Permission state | Distinguishes selection from authorization to act |
| Application and configuration version | Identifies the behavior under test |
| Selected operation and result | Shows the actual mismatch |
| Expected handling | Defines a correct selection or a needed clarification |
If current time affects the task, record the relevant clock and timezone assumption. If a dependency must remain live, state that the case is less controlled and retain the response observed in each run.
Keep a neighboring case
A rule that always asks for clarification might avoid selecting the wrong appointment while making ordinary cases unnecessarily difficult. Add a case with one unambiguous match and another with no match.
Judge the requested outcome and important constraints separately. The case should reveal whether the fix resolved the selection problem, not merely whether the assistant stopped making any proposal.
Separate fixture repair from application improvement
Sometimes the original evidence is incomplete. Mark the missing field and build a controlled reproduction rather than pretending the production event is fully reconstructable. If the expected result was wrong, record a fixture correction independently of the candidate’s result.
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. The trace investigation guide connects recorded operations with prepared evaluation data. The eval-driven guide explains how a reviewed failure becomes part of release review. OpenTelemetry’s trace concepts provide a foundation for the recorded-operation side of this work.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.