Write a Reviewable Hypothesis for an Agent Failure
Shashank Agarwal
A persuasive explanation is easy to write after an agent fails. A useful hypothesis is one that another engineer can inspect and test.
Quick answer: State the observed mismatch, the proposed mechanism and the evidence supporting it. Name an alternative explanation and a controlled test that could contradict the hypothesis. Keep confidence separate from severity: an important failure can still have an uncertain cause.
Avoid a diagnosis that explains everything
For an illustrative address assistant, “the model ignored context” is too broad. A narrower hypothesis is: “The operation used the earlier postal code because the context builder omitted the user's correction.”
This claim predicts an inspectable artifact: the correction should be missing from the context supplied before the operation.
Use a hypothesis card
| Field | Example |
|---|---|
| Observation | Final operation used the earlier postal code |
| Mechanism proposed | Correction omitted during context assembly |
| Supporting evidence | Captured context lacks the later message |
| Alternative | Tool argument mapping replaced the selected value |
| Discriminating test | Compare supplied context and mapped arguments |
| Open limit | Original context may be incompletely captured |
These are synthetic examples. They do not describe a private trace or a measured Noveum incident.
Try to reject the explanation
If the correction is present in the supplied context, the omission hypothesis needs revision. Inspect argument selection and mapping rather than redefining the original hypothesis until it fits.
A passing rerun after a prompt change does not automatically prove the prompt caused the failure. The input, fixture or external state may also have changed.
Connect the hypothesis to a bounded task
A reviewable task names the proposed change, affected cases and evidence required before closure. Keep alternative explanations in the investigation record until they are resolved or explicitly deferred.
The purpose is not to sound certain quickly. It is to make the next engineering action test a specific explanation.
I'm Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. These are proposed engineering methods and illustrative fixtures, not measured customer results.
Use the tool-boundary guide for operation evidence and the regression-test guide for controlled verification.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.