Voice Agent Test Scenarios: 9 Cases Manual Testing Misses
Pragati Tripathi
The scenarios that break voice agents in production are rarely the ones nobody thought of. They are more like: a caller already frustrated at turn one, a market at a specific noise level, an interruption landing 400ms into a sentence. Thinking of those calls is easy. Producing one, at a set intensity, more than once, is what a person cannot do.
Which leaves a question no amount of manual testing answers. The agent works for the ten callers a team could represent. Does it hold for the ten thousand it will actually meet in the real world? This blog answers how and why to evaluate your voice AI with simulated real-world callers.
Why a voice test you cannot rerun is not a regression test
A person can always call again. What a person cannot do is set the variables that decide the call.
Consider a colleague calling from a train platform at rush hour, talking over the agent twice because she is distracted. The agent speaks through both interruptions and answers the wrong question. That is a real bug, found honestly.
Reproducing it is where the process breaks. Not because she cannot call back, but because she cannot call back the same way: the platform noise at the same level, the interruptions at the same two points in the agent's sentence, the same distraction. She also knows she is testing now, which means she is placing a call rather than receiving one. A tester performing frustration is not a caller who is frustrated.
So the fix cannot be proven. Three prompt changes later, someone adjusts the system prompt, the same barge-in behaviour returns, and no test fails.
A scenario tested once lives in someone's memory, not in a regression suite. That gap is not a discipline problem. It is a property of voice, where the conditions are as much a part of the test as the words are.
Why manual voice AI tests aren't reproducible
Five reasons explain why manual voice AI testing is a red flag in 2026. As voice agents tackle increasingly complex tasks, knowing where they break and why tells you whether the answer is simulation, better tooling, or simply more hours than anyone has.
Timing precision
An interruption 400ms into the agent's sentence and an interruption 900ms in hit different code paths. The first tests your voice activity detection threshold. The second tests turn-detection logic after the agent has committed to a response. A human tester cannot target 400ms deliberately, and cannot hit the same mark twice in a row. The variable that decides the outcome is the one you have least control over.
Acoustic reproducibility
Background noise is a dial, not a switch. Street noise at level 15 and street noise at level 50 are different tests with different failure modes, and the second is where speech recognition starts dropping syllables. Standing in an actual market gives you one unlabelled point on that dial, once, with no way to return to it.
Caller state
"Already frustrated before the agent speaks" is a starting condition, not an event. A colleague acting frustrated is performing, and performing differently on take four than on take one. What you are measuring is how frustration moves across turns, and a performance cannot hold that steady across forty runs.
Network conditions
Packet loss, jitter and codec degradation on a live call are not yours to set. You cannot induce a four-second lag on a real phone connection on request, which means the failure mode your users hit on bad mobile data is one you have never seen.
Combinatorics
Eight personas, six conditions, five intents. That is 240 calls. At three minutes each it is twelve hours of dialling, and it is twelve hours again after the next prompt change. Check the arithmetic against your own matrix. The reason nobody runs it is not laziness.
Producing a voice scenario and grading conversational AI are different problems
Some caller scenarios are trivial to produce and hard to grade.
Play a recording where the agent says a medication name wrong. You staged that in ten seconds. Now, what told you it was wrong? Your audio dashboard shows a healthy MOS score and no alerts fired, because it is measuring clarity and noise, not whether the word was correct. The test ran and taught you nothing.
Others run the other way. A four-second network lag is hard to produce and trivial to grade. Either the agent handled the silence or it hung up.
The fixes are different. Simulation fixes staging a real-world caller; scoring fixes regression testing of voice AI agents. A team that invests in a voice AI simulation platform to solve a scoring problem runs five hundred calls, reads five hundred scored passes, and ships with more confidence but the same bug.
The Stage / Score Grid for simulating voice AI scenarios
Every caller scenario sits in one of four places. Find yours, and the quadrant tells you whether the gap is in producing the call or in grading it.
| Easy to score | Hard to score | |
|---|---|---|
| Easy to stage | Q1. Your current suite. "What are your opening hours?" Anyone can dial it and pass or fail is obvious. Most existing test plans sit almost entirely here. | Q2. The false pass. The agent mispronounces a surname. You produced it in seconds and nothing graded it. The test reports green. |
| Hard to stage | Q3. The known unknown. Four-second network lag. You would know instantly whether the agent survived, if you could produce it reliably. | Q4. Ships broken. Frustrated caller in a crowded market who interrupts twice and changes her mind. You cannot produce it, and could not grade it if you did. |
The diagnosis takes ten seconds. Q2 means you have an evaluation gap and buying a simulator will not help. Q3 means you have a simulation gap and your existing metrics are fine. Q4 needs both fixed, which is why it is the quadrant that reaches production.
Nine hard voice scenarios, and the scorer that catches each one
Nine scenarios worth building before running your voice AI in production sit below. Each row names the caller condition, the variable a person cannot set to reproduce it, and the Noveum scorer that catches the failure when it happens.
| Caller scenario | Quadrant | Variable a person cannot set | Scorer that catches the failure |
|---|---|---|---|
| Caller interrupts 400ms into the greeting | Q4 | Timing precision | user_interrupt_agent, assistant_overlap |
| Caller half-whispers in a hospital corridor | Q4 | Acoustic reproducibility | stt_over_suppression, word_accuracy |
| Caller is frustrated before turn one | Q4 | Caller state | sentiment_csat, drop_off_node |
| Network drops half a syllable every few seconds | Q3 | Network conditions | audio_breakage, e2e_latency |
| Caller changes their mind mid-sentence | Q4 | Caller state, combinatorics | intention_fulfillment |
| Agent mispronounces a medication or surname | Q2 | Stageable. Nothing grades it. | mispronunciation, pronunciation_audio |
| Long silence while the caller finds their card | Q3 | Timing precision | assistant_silence, on_user_turn_completed_delay |
| STT, LLM and TTS all run slightly slow at once | Q3 | Network, timing | e2e_latency, tts_ttfb |
| Agent invents a policy under a persistent caller | Q2 | Stageable. Audio metrics cannot see it. | hallucination_detection, is_harmful_advice |
Three of these sit in Q2. You can already produce all three, no simulation tool improves them, and the gap is entirely in grading. For the mispronunciation row, a person listening is still the most reliable check you have.
Interruption runs in two directions, and Noveum counts them separately. ai_interrupt_user counts agent barge-ins, where the agent starts speaking while the caller's utterance is still open. user_interrupt_agent counts caller barge-ins, and carries stop_time_after_interruption_s in its metadata: how long the agent kept talking after the caller cut in. Alongside those, assistant_overlap measures dual-talk duration from Silero VAD rather than counting events.
Counting and duration answer different questions. An agent that interrupts once for four seconds and one that interrupts four times for a quarter-second each produce the same event count and a very different call.
Listen to a voice agent missing an interruption at a crucial time
Every page ranking for voice agent testing describes these failures in prose and tables. None of them lets a reader hear one. Post 3_ Hard to generate scenarios.mp4Download
Why a synthetic caller written just as a script fails voice AI testing
Two ways exist to define a synthetic caller to rigorously test your voice AI, and the choice decides whether the scenario can be rerun.
As a script. You write an instruction: "Call the customer to propose upgrading their liability insurance and explain the benefits." A language model plays that caller. It is fast to write and it reads naturally.
As a parameter set. You set values: interruption rate 0.70 on a 0 to 1 scale, speaking speed 1.00 on a 0.7 to 1.2 scale, background noise "street" at level 20 on a 5 to 90 scale, primary languages English and Hindi, a specific synthesis voice ID, a named TTS model.
Run the script version twice and the two calls differ in ways nothing recorded. The model interrupted more on one run than the other, and no field says by how much, so there is nothing to hold constant when you rerun it.
Run the parameter version twice and every variable that shaped the call is written down and unchanged. The caller comes back at the same interruption level, the same speed, the same noise profile. A prompt gives you a plausible call. A parameter set gives you a call you can put in a suite.
How voice agent testing platforms handle simulation and scoring
Two axes decide what a platform can do for you: whether it can produce the call, and whether it can grade the audio once produced. This table places five options on both.
| Stages real audio calls | Scores the audio itself | Scores transcript behaviour | Caller defined as | Ships a fix | |
|---|---|---|---|---|---|
| Hamming | Yes | Yes | Yes | Scenario definition | Reports the failure |
| Cekura | Yes | Yes | Yes | Persona plus instructions | Reports the failure |
| Coval | Yes | Yes | Partial | Natural-language simulator prompt | Reports the failure |
| Text-first platforms (Arize Phoenix, Langfuse, Braintrust) | No | No | Yes | n/a | Reports the failure |
| Noveum | Yes | Yes | Yes | Numeric parameter set | Recommends a change, backtested on the failing calls |
How to choose the first 20 voice scenarios for regression testing
The published advice in this category is a number. Five hundred scenarios. Ten thousand scenarios. Eighteen pre-built. None of it says which ones.
Two steps, and most teams need only one of them. If a list of candidate scenarios already exists, skip to step two.
Step one: Which callers has this agent never met?
Teams stuck here are not short on effort. They are short on a way to picture callers who have not called yet, which is difficult to do from a blank page and easy to do from a list.
Start from the persona library rather than imagination. Each of the 255 personas carries an accent, a speaking speed, an interruption rate and a background noise profile as settable values. Read down it and mark every persona your agent has never handled. That list is the candidate set, and it takes about twenty minutes.Three sources sharpen it, in order of how much they cost:- The persona library, filtered to the languages and regions your callers actually come from
- Support tickets and complaint logs, which name the conditions real customers described even when no recording exists
- Call recordings, if there are any. Teams evaluating Noveum often send transcripts before anything is integrated, precisely to find out what a scorer catches that a human review missedNone of the three requires production traffic. The second and third sharpen the list. The first produces it.
Step Two: Which twenty go first?
Score each candidate on three factors, one to three each, and multiply.
Blast radius. What it costs when it fails. A mispronounced medication name on a healthcare line is a 3. A slightly slow greeting is a 1.
Frequency. How much of your real traffic hits this condition. Read your call logs, not your intuition. Most teams overrate accents and underrate silence.
Stage difficulty. Where it sits on the grid. This one runs backwards from instinct. The hardest-to-stage scenarios score highest, because they are the ones your manual process structurally cannot reach. Anything you can already dial is already covered.
Worked example, healthcare IVR. Mispronounced medication name: blast radius 3, frequency 2, stage difficulty 1 because you can produce it trivially. Score 6. Frustrated caller in traffic who interrupts: blast radius 2, frequency 3, stage difficulty 3. Score 18. The second one goes first, even though the first one sounds scarier, because the second is the one your current process will never catch on its own.
Rank, take the top twenty, stop. Twenty scenarios you rerun after every prompt change beat five hundred you ran once. The second number looks better in a deck and does less for you.
Where simulation ends and scoring your voice AI agent begins
NovaSynth moves scenarios leftward on the stage axis. A synthetic caller defined by numbers rather than description, carrying interruption level, speaking speed, background noise type, level and duration, accent, language set and a specific synthesis voice, calls your agent over a real voice connection: SIP for phone, WebRTC for LiveKit. It is a call, not a replayed transcript, so it exercises speech recognition and codec handling rather than skipping them. The shared library holds 255 personas, and it connects to LiveKit, Pipecat, LangChain, LangGraph and CrewAI.
The scorer library moves scenarios downward on the score axis. 100+ scorers, 25+ of them for voice and audio, grading the call itself rather than a transcription of it.
Both axes, because closing one leaves the other standing.
Neither closes the last gap. Simulation cannot tell you whether a response sounds trustworthy to a human being. Warmth, hesitation, whether the agent sounds like it is reading off a page. A person listening to twenty calls still catches things no scorer does, so budget the hour.
The number that matters is not how many voice scenarios you have
The real number is how many could run again tomorrow, identically, and return a comparable answer.
Count the rows in your test plan that pass that test. For most teams the figure is smaller than expected, and that figure is the true size of the regression suite.
We are launching NovaSynth on Product Hunt on 3 September. It is built for the left half of that grid: synthetic callers with real personas, over a real telephony connection, so the scenarios you cannot stage become scenarios you can rerun. Do share your feedback when we go live.
Questions engineers ask about testing voice scenarios
Our agent works in testing but breaks when real users hit it. How do I close that gap?
Your test environment holds constant the four variables that break agents in production: audio quality, caller behaviour, language variation and background noise. Real callers vary all four at once. Closing the gap means running tests under those conditions, which for most teams means synthetic callers rather than more manual calls.
How do I run regression tests every time my voice agent prompt changes?
Convert each scenario into a fixture that can be replayed identically: a fixed persona, a fixed scenario definition, and a fixed set of scorers with thresholds. Run the same batch before and after the change and compare per-scorer results, not just pass counts. A prompt edit that changes wording without changing behaviour should pass, so compare on scorer output rather than transcript text.
How can I turn failed production voice calls into regression test cases?
Take the trace of the failing call, extract the caller conditions that produced it, and rebuild those as a persona and scenario. The failure becomes reproducible, which is what makes it a test rather than an incident report. It is the only path from "we saw it once" to "it cannot come back."
What role do silence detection metrics play in voice agent quality evaluation?
Silence is where callers decide the line is dead. assistant_silence measures dead-air as a fraction of the whole call, and it is deliberately non-monotonic: a moderate silence ratio scores highest, because an agent that never pauses is as unnatural as one that pauses too long. For the specific gap after a caller stops talking, on_user_turn_completed_delay measures the turn-completion callback delay directly. Neither word accuracy nor call completion rate exposes either one.
What tools can measure voice agent audio quality in production?
Anything scoring the audio rather than the transcript. In Noveum that is the audio-tagged scorers: audio_breakage for dropouts, word_accuracy for transcription degradation under noise, mispronunciation for wrong pronunciations of names and terms, stt_over_suppression for noise reduction eating quiet speech, and assistant_overlap for how long agent and caller spoke at once. Text-first evaluation platforms cannot do this, because they receive a transcript and the audio problems are already gone by then.
How do I set up evaluation for a voice agent in production?
Trace every call, build a dataset from real traffic, run scorers against it on a schedule, and gate deploys on the scores. Tracing has to come first, because you cannot evaluate what you never captured, and a dataset built from real calls beats one written from imagination.
If my voice agent mispronounces names or talks over customers, which tools catch that, and how do they differ from text-based eval tools?
Both failures are invisible to text-based tools by construction. A transcript records the word the agent meant, not the sound it made, and it records who spoke without recording that they spoke simultaneously. Catching them needs scorers that read the audio: mispronunciation and pronunciation_audio for the first, ai_interrupt_user and assistant_overlap for the second.
How do I compare two voice agent versions on latency and compliance?
Run the same scenario batch against both versions and diff per-scorer. For latency, compare on e2e_latency rather than any single component, because what a caller experiences is the sum of turn detection, transcription, time to first token and time to first byte of speech. Each of those has its own scorer, so a regression can be traced to the stage that caused it. Optimising one component while another degrades looks fine per-component and worse on the call. For compliance use instruction_adherence and is_harmful_advice, which catch constraint violations audio metrics cannot see. The stack-specific setup on LiveKit is in our LiveKit evaluation guide.
Related: LiveKit Voice Agent Evaluation · Voice Agent Evaluation in Production · Production Regression Testing · The Complete Guide to Voice AI Evaluation · Scorer library · NovaSynth docs
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.