The Baseline an AI Improvement Claim Needs
Shashank Agarwal
An improvement claim needs a baseline that another reviewer can understand. “Better than before” leaves both the earlier system and the comparison undefined.
Quick answer: Preserve the baseline and candidate configurations, comparable cases, evaluator version and execution conditions. Define the metric and report relevant regressions. Attribute the result to the tested scope rather than treating a selected success as proof of universal improvement.
Name what changed
For an illustrative scheduling agent, the candidate may change how a corrected date updates pending state. The baseline should identify the earlier prompt and workflow, not merely a model family name.
If the tool contract also changed, record that difference.
| Baseline item | Why it matters |
|---|---|
| Agent configuration | Identifies the system being compared |
| Dataset | Defines the tasks represented |
| Fixtures | Controls relevant external state |
| Evaluator | Defines how results are judged |
| Attempts | Preserves variation and failures |
| Metric | Makes the comparison interpretable |
This is a proposed review framework, not a measured Noveum benchmark.
Keep the comparison fair
Do not compare the baseline's hardest cases with a candidate's selected successes. Preserve the same meaningful test set where possible and report new coverage separately.
If conditions cannot be held stable, explain the limitation rather than assigning all observed difference to the candidate.
Report the countervailing result
A candidate can improve corrected-date cases while making ordinary bookings ask unnecessary questions. Review both groups before publishing a broad quality claim.
A summary should lead to the changed cases and underlying evidence.
Make the next claim bounded
“The candidate met these acceptance conditions under this fixture” is more useful than an unsupported general reliability statement.
See the run-comparison guide and evaluation-report guide for the artifacts behind the baseline.
A good baseline makes a result easier to challenge and therefore easier to use in a real decision.
I'm Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. Follow my work on LinkedIn. Drafted with AI assistance.
Get Early Access to Noveum.ai Platform
Join the select group of AI teams optimizing their models with our data-driven platform. We're onboarding users in limited batches to ensure a premium experience.