Zero Failures in a Voice-Agent Test: How Many Calls Are Enough?

Shashank Agarwal

#Voice Agent Evaluation#Release Testing#Statistics#Python

A voice agent passes 50 calls in a release test. How rare could a specific failure be, given what you observed? Under a simple binomial model, 50 calls with zero failures give a one-sided 95% upper bound of about 5.82%. To put that bound at or below 1% with zero failures, the same model requires 299 calls. The result depends on what counts as a failure and which calls belong in the sample.

This guide defines one release criterion, shows the arithmetic, and gives you a calculator that runs without credentials or third-party packages. All call counts and results are illustrative.

Define the event before you run the calls

Imagine an agent that routes customers to billing support. Its job is to offer a transfer after it recognizes a billing dispute. Your test can define failure as any eligible call where the agent ends without offering that transfer. Write down how to handle disconnected calls, transcription errors, and ambiguous recordings before looking at outcomes.

The call is the experimental unit. One call can contain several turns. Counting every turn as an independent trial gives the sample an artificially large denominator. A batch of repeated synthetic calls can expose a reproducible bug. If the same caller fixture, seed, or shared outage makes the outcomes dependent, the uncertainty calculation needs to account for that dependence.

Keep a manifest that gives each eligible case an ID and lists its expected outcome. Log why a case received no score. A judge that never saw a recording leaves an unresolved result. A call where the product disconnected can be an observed failure if the rubric identifies it that way. For a runnable example of reporting coverage, see When Agent Evaluation Scores Are Missing.

Translate zero observed failures into an upper bound

Let n be the number of independently scored calls from a specified test population, and let p be the probability of the failure you defined. If each call has the same marginal failure probability and the calls are independent, the probability of seeing zero failures is (1 - p)^n. At the one-sided 95% upper limit, that probability is 0.05. Solving the equation gives p_upper = 1 - 0.05^(1/n). NIST exact binomial confidence limits

One way to read the result is to imagine repeating the 50-call procedure with a true failure rate of 5.82%. Approximately one run in 20 would still produce zero failures. Larger failure rates make a clean batch even less likely under the model.

Independently scored callsObserved failuresOne-sided 95% upper bound
20013.91%
5005.82%
10002.95%
30000.99%
60000.50%

The bars show the corresponding upper bounds. Each # represents roughly half a percentage point. The labels give the rounded values.

text
 20  ############################  13.91%
 50  ############                   5.82%
100  ######                         2.95%
300  ##                             0.99%
600  #                              0.50%

For a test population and a target rate you select in advance, invert the equation to estimate the required number of calls. A 95% upper limit of 5% or less requires 59 clean calls. A 1% target requires 299 clean calls. A 0.5% target requires 598 clean calls.

Calculate the bound and sample size in Python

Save the following as zero_failures.py and run python3 zero_failures.py. Pass probabilities as fractions, such as 0.01 for 1%. The functions and chart use the Python standard library.

python
from math import ceil, expm1, log1p


def log_tail(confidence):
    if not 0 < confidence < 1:
        raise ValueError("confidence must be between 0 and 1")
    return log1p(-confidence)


def upper_bound_after_zero(calls, confidence=0.95):
    if type(calls) is not int or calls <= 0:
        raise ValueError("calls must be a positive integer")
    return -expm1(log_tail(confidence) / calls)


def calls_required_after_zero(target, confidence=0.95):
    if not 0 < target < 1:
        raise ValueError("target must be between 0 and 1")
    return ceil(log_tail(confidence) / log1p(-target))


if __name__ == "__main__":
    for calls in (20, 50, 100, 300, 600):
        bound = upper_bound_after_zero(calls)
        bar = "#" * round(bound / 0.005)
        print(f"{calls:3}  {bar:28}  {bound:6.2%}")

    for target in (0.05, 0.01, 0.005):
        calls = calls_required_after_zero(target)
        assert upper_bound_after_zero(calls) <= target
        assert calls == 1 or upper_bound_after_zero(calls - 1) > target
        print(f"Target {target:.1%} requires {calls} calls")

Use the first function only for zero observed events. If there are observed failures, use an exact binomial interval that takes the observed count as well as the number of trials. Keep independent coverage checks alongside the calculation so missing outcomes remain visible.

Decide what the release test can establish

If missed transfers matter across languages, plan how to sample each language. A holdout containing mostly English calls can leave a smaller language group weakly tested even if the pooled sample is large. Report per-group denominators and results. Treat a set of per-group confidence bounds as separate claims unless you account for simultaneous coverage.

Build the release decision around the failure criterion, the call populations, and the design you set before testing. Use varied account states and distinct caller scenarios for evidence about typical traffic. Use deliberately repeated adversarial or edge-case scenarios to reproduce known bugs. Report those targeted results separately from a probability statement about an intended population.

Keep the planned sample size fixed before reading the outcome. Repeatedly checking a fixed-sample bound and stopping at the first favorable result changes the procedure. If a release genuinely needs continuous checks, use a sequential design with an explicit stopping rule. Record version, rubric, sampling approach and whether any outcomes were unscored so another engineer can reproduce the decision.

If the available sample cannot meet your target, document what it supports and use the gap to decide what to test next. In the example, 50 clean calls give useful release evidence about frequent failures. A claim about a 1% failure rate calls for a larger independent sample or a decision with more limited scope.

Shashank Agarwal is the founder and CEO of Noveum AI. This guide was prepared with AI assistance. The counts are a synthetic example and the bounds come from the stated binomial model.

Exclusive Early Access

Get Early Access to Noveum.ai Platform

Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.

Early access members receive premium onboarding support and influence our product roadmap. Limited spots available.

Zero Failures in a Voice-Agent Test: How Many Calls Are Enough? | Noveum.ai