When Agent Evaluation Scores Are Missing, Check Coverage First

Shashank Agarwal

#Agent evaluation#Evaluation coverage#Python#Release testing

An agent’s pass rate can rise when its hardest cases stop receiving scores. Before signing off on a release, check how many cases entered the evaluation, how many received a result, and which cases went missing. Keep the missing cases in the report even when the scorer has nothing to say about them.

This walkthrough uses a 24-case synthetic support-agent fixture. The inputs and results are illustrative and can be reproduced with Python’s standard library.

Start with the cases you intended to evaluate

Suppose a support team runs two versions on the same 16 routine requests and eight escalation requests. Both versions record 15 passes. The baseline has 21 scored cases. The candidate has 17 because scores are missing from additional escalations.

VersionIntendedPassFailUnscoredScoring coveragePass rate among scored
Baseline24156387.5%71.4%
Candidate24152770.8%88.2%

The candidate has the same number of confirmed passes and fewer observed failures. Its overall pass rate among scored cases is up 16.8 percentage points. The coverage drop tells you why the headline alone is insufficient. Baseline escalation coverage is 6 of 8. Candidate escalation coverage is 2 of 8. Routine coverage stays at 15 of 16.

The bars show each version’s 24 case outcomes. Each character represents one case. P means pass, F means fail, and ? means unscored.

text
baseline   PPPPPPPPPPPPPPPFFFFFF???
candidate  PPPPPPPPPPPPPPPFF???????

In your pipeline, build the evaluation report from a manifest of intended case IDs. Left-join outcomes to that manifest on both case ID and version. A report built only from scorer outputs can never see a case that produced no output. Make the manifest first, with the expected scenario and rule version recorded before the test runs. Use a stable case ID for the comparison and keep the run or trace ID alongside each attempt. OpenTelemetry’s context propagation guide

Show how much the missing outcomes could change the result

You can calculate a range for the intended cohort without guessing how its missing scores would turn out. Divide confirmed passes by all intended cases for the lower endpoint. For the upper endpoint, add the unscored cases to the numerator. The range is conditional on the recorded binary results being accurate. It describes extreme assignments for this specific cohort and carries no confidence level.

VersionAll unscored cases failAll unscored cases pass
Baseline15/24 = 62.5%18/24 = 75.0%
Candidate15/24 = 62.5%22/24 = 91.7%

In this fixture, several interpretations remain possible. The candidate might have fewer genuine failures, or it might fail on escalations that were never scored. The bounds overlap, so a release reviewer has a concrete task. Recover missing escalation outcomes and rerun both versions with the same scorer contract.

If an agent failed to produce a deliverable and the rubric says that is a failure, record it as a failure. Use unscored when the team lacks evidence to apply the rubric. For example, an exporter timeout can prevent a grader from seeing an otherwise completed request. Give timeout, parse error, and missing trace their own reason codes. A terminal application failure that the test deliberately measures belongs in the failure count.

Reproduce the report in Python

Save the code below as coverage_report.py and run python3 coverage_report.py with Python 3.10 or later. The manifest fixes the cohort and scenario labels. In a real export, replace the fixture dictionaries with the final results for each version. A missing key represents an unscored case. The validator rejects unknown case IDs and result values before reporting.

python
from collections import Counter


def ids(prefix, count):
    return [f"{prefix}{number:02d}" for number in range(1, count + 1)]


def outcomes(case_ids, statuses):
    return dict(zip(case_ids, statuses.split(), strict=True))


# Explicitly assign every case that received a score.
routine, escalation = ids("R", 16), ids("E", 8)
manifest = routine + escalation
baseline = {
    **outcomes(routine[:15], "pass " * 12 + "fail " * 3),
    **outcomes(escalation[:6], "pass " * 3 + "fail " * 3),
}
candidate = {
    **outcomes(routine[:15], "pass " * 13 + "fail " * 2),
    **outcomes([escalation[0], escalation[3]], "pass pass"),
}


def validate(manifest, results):
    if len(set(manifest)) != len(manifest):
        raise ValueError("duplicate case ID in manifest")
    if set(results) - set(manifest):
        raise ValueError("score contains a case outside the manifest")
    if set(results.values()) - {"pass", "fail"}:
        raise ValueError("score contains an unknown outcome")


def percentage(numerator, denominator):
    return f"{numerator / denominator:.1%}" if denominator else "n/a"


def summarize(manifest, results):
    validate(manifest, results)
    counts = Counter(results.get(case_id, "unscored") for case_id in manifest)
    passed = counts["pass"]
    failed = counts["fail"]
    missing = counts["unscored"]
    total = len(manifest)
    coverage = percentage(passed + failed, total)
    pass_rate = percentage(passed, passed + failed)
    lower = percentage(passed, total)
    upper = percentage(passed + missing, total)
    bar = "P" * passed + "F" * failed + "?" * missing
    return counts, coverage, pass_rate, (lower, upper), bar


def main():
    for version, results in [("baseline", baseline), ("candidate", candidate)]:
        _, coverage, pass_rate, bounds, bar = summarize(manifest, results)
        escalation_scored = sum(case_id in results for case_id in escalation)
        print(f"{version:9}  {bar}  coverage {coverage}, scored pass {pass_rate}")
        print(f"{'':11}range {bounds[0]} to {bounds[1]}")
        print(f"{'':11}escalation coverage {escalation_scored}/{len(escalation)}")

    pairs = Counter(
        (baseline[case_id], candidate[case_id])
        for case_id in baseline.keys() & candidate.keys()
    )
    print(f"{pairs.total()} cases scored in both versions")
    print(f"fail to pass {pairs['fail', 'pass']}, pass to fail {pairs['pass', 'fail']}")


if __name__ == "__main__":
    main()

The paired view gives you one additional diagnostic. Seventeen cases have both scores. Two of those cases move from fail to pass and zero move in the other direction. That is evidence about those cases. The seven cases outside the intersection still need investigation, especially because six missing candidate cases are escalations. A common-case comparison can help distinguish real changes from a shift in which cases are scored, while leaving that coverage problem visible.

Decide what blocks the release

Write the minimum coverage you need before running the suite. A team might require every safety-critical or escalation fixture to receive a final result, with a separate minimum coverage threshold for the larger routine cohort. Report distinct counts at each stage such as intended, executed, traced, eligible, and scored. Those counts let you tell a scheduling issue from a capture failure or a grader rejection.

Keep attempted runs when retrying. Associate the chosen score with the original case and the recorded attempt. Choose the selection rule before inspecting the outcomes. Report both versions with the same report window and the same rule. If the job can finish asynchronously, set a common cutoff so one version doesn’t get longer to catch up. For an example of why monitoring evidence and task outcomes belong side by side, see the production monitoring guide.

After the rerun, review the cases that changed from pass to fail as well as the opposite changes. Ask the person responsible for the affected workflow to check a small sample of disagreements against the rubric. Document the remaining unknowns in the release decision. In the fixture above, a reviewer can name the missing group and ask for a concrete rerun before treating the higher scored pass rate as release evidence.

Shashank Agarwal is the founder and CEO of Noveum AI. This guide was prepared with AI assistance. All example results come from the synthetic fixture.

Exclusive Early Access

Get Early Access to Noveum.ai Platform

Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.

Early access members receive premium onboarding support and influence our product roadmap. Limited spots available.

When Agent Evaluation Scores Are Missing, Check Coverage First | Noveum.ai