How to Write a Voice Scenario That Can Fail
Shashank Agarwal
A scenario that can only end with a friendly completed conversation is a weak test. Define the conditions under which the voice agent should succeed, clarify, stop or report a limitation.
Quick answer: Write a goal, controlled starting state and observable acceptance conditions. Add a branch that can reveal a meaningful failure, then retain the evidence needed to distinguish agent behavior from a fixture or transport problem. A successful call connection is not the task’s acceptance criterion.
Build the failure into the decision
Imagine an illustrative delivery-change call. The caller requests a day the controlled fixture does not offer. The expected behavior is to explain availability and ask whether an offered alternative is acceptable.
The test can fail if the agent invents availability, applies an unconfirmed alternative or claims a change that the action record does not establish. It can also fail if the agent ignores a valid option and needlessly ends the interaction.
Keep the acceptance conditions explicit
| Test element | Example definition |
|---|---|
| Starting state | Existing delivery and controlled available days |
| Caller goal | Move the delivery to the requested day |
| Unavailable branch | Explain the gap and offer only supported alternatives |
| Confirmation | Preserve the caller’s approval of an actual proposal |
| Action evidence | Establish what change, if any, was applied |
| Final explanation | Describe the observed result accurately |
This is a synthetic test design. It does not describe a measured Noveum run or a live delivery integration.
Avoid a simulator that supplies the answer
Do not let the synthetic caller reveal information the real caller would not possess simply to keep the conversation moving. If the caller is meant to know only an order reference, the test should not quietly give it the entire backend state.
Keep the simulator instructions and agent instructions separate in the test record. Otherwise a later reviewer may be unable to tell which participant introduced an unsupported assumption.
Review why the run failed
A disconnected call, a missing fixture and an incorrect action are different results. Retain each condition rather than forcing all runs into a single quality score. Repeat a case only under a stated method, and preserve the previous result instead of replacing it with the first pass.
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. NovaSynth is part of the configured-scenario testing workflow. The team still defines the requirement and reviews the evidence.
The repeatable-failure guide explains fixtures and starting state; the tool-boundary guide explains action evidence.
Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.