Composite
Overall ranking
Equal-weighted average across five comparable dimensions. Balance beats winning one column.
Full section →
Coming soonVoice AI Platform Benchmarking
The 2026 AI Eval Platform
Benchmarks
Which AI eval platform actually catches failures without flagging good answers?
Section 01 –Explore
What we found
Scroll through each dimension. Five cards, one story.
Faithfulness
Recall vs calibration
Top-left on the scatter wins. Catch failures with high recall without flagging good answers.
Full section →Scorer trap
Opposite verdicts
The scorer you enable can matter more than the vendor you choose.
Full section →Latency
Judge speed
Noveum returns a verdict in 0.59s, 2 to 27 times faster than the field.
Full section →Consistency
Cross-dimension balance
Bright, even rows beat tall peaks and holes. Consistency is the rarest property in the study.
Full section →| faithF1 | ansRel | ctxRel | toolSel | |
|---|---|---|---|---|
| Noveum | 1.00 | 1.00 | 0.99 | 1.00 |
| DeepEval | 0.72 | 0.05 | 0.97 | 1.00 |
| Arize / Phoenix | 0.87 | 0.08 | 0.91 | 0.89 |
| Galileo | 0.92 | 0.00 | 0.83 | NA |
Section 02 –Methodology
How it works
A real agent runs adversarial conversations. Each platform scores with its own native scorers, no cheat sheet. Five steps from stress test to published, reproducible scores.
⬡
RAG agent
LangGraph chatbot on gpt-4.1-mini over 599 documentation chunks. Grounded and deliberately ungrounded variants.
›
❯_
100 conversations
440 assistant turns · 131 genuine hallucinations · 9 adversarial scenario families
›
⦿
Opus-4.8 referee
Claude Opus-4.8 labels every turn. Never shown to any platform.
›
▦
8 native scorers
Reference-free · each platform's documented best setup
›
∎
Published scores
Bootstrap 95% CIs, 2,000 resamples · fully re-derivable
Noveum's method beat ClosedQA on the identical GPT judge: 82% vs 55% recall. The difference is the method, not the model.
◆
Real agent
LangGraph RAG chatbot on gpt-4.1-mini. Skips retrieval on 53% of turns. 78% of its hallucinations came with no retrieved context at all.
⦿
Neutral referee
Claude Opus-4.8 labels all 440 turns for hallucination, quality, relevance, and role adherence. Never an input to any platform.
≋
Statistics, not vibes
Bootstrap 95% CIs on every rate. Paired significance tests on close calls. Ties reported as ties.
Section 03 –Composite
Overall ranking
Who scores best across all dimensions on average? The composite is an equal-weighted average of the five dimensions that share a comparable metric. Balance beats winning one column.
Hosted-only scorers are excluded from a platform's average rather than counted against it.
Clear separation at the summit.
Noveum 0.999 vs DeepEval 0.650. The composite gap at the top is decisive.
Balance beats one dimension.
Only Noveum scores high across all five composite dimensions at once.
No platform owns every column.
Ragas leads context-relevance. Braintrust leads faithfulness false-alarm rate. Arize has the best-balanced tool-argument profile. Noveum's headline is balance.
Section 04 –Faithfulness
Recall vs calibration
Faithfulness balances hallucination recall against false-alarm rate. For unsupervised gating, false alarms decide usability. Noveum leads on balance at 89% recall and 10% false-alarm; Ragas catches more but flags 37% of good answers.
01. Best balance
Highest F1 and strongest quality correlation across faithfulness scorers.
F1 composite84%
02. Calibration leader
Braintrust ClosedQA posts the lowest false-alarm rate in the study.
False-alarm rate8%
03. High recall, unusable
Ragas catches the most hallucinations but flags a third of good answers.
Hallucination recall94%
04. Purpose-built, missed
Patronus Lynx scores lowest on conversational fabrications it was built for.
Fabrication recall48%
The top three recall scores (Noveum, Braintrust, Arize) overlap within their confidence intervals. Under the stronger paired test, Noveum's edge over Braintrust is significant while its edge over Arize is borderline. We report the cluster as a statistical tie on purpose.
Full report
Unlock the complete benchmark
Go deeper with the 22-page report and buyer's guide — six job-to-shortlist maps built directly from the data.
Section 05 –Scorer trap
Same data, opposite verdicts
The scorer you enable can swing results more than the vendor you pick. On identical turns, Braintrust swings 58 points between scorers; DeepEval's dedicated metric catches 1% vs 73% for GEval. Audit which scorer is actually enabled.
58-point swing on identical turns.
Dedicated metric: 1%. GEval: 73%.
High recall, nearly half of good answers flagged.
01. Scorer swing
Same vendor, same data — opposite verdicts from scorer choice alone.
False-alarm delta58 pts
02. Metric mismatch
FaithfulnessMetric catches almost nothing; GEval catches 73% on identical turns.
Recall delta72 pts
03. Over-flagging
Maxim catches nearly everything but flags almost half of good answers.
Good-answer flags48%
Section 06 –Failure map
Specialization patterns
Aggregate recall hides specialization. Fabricated code APIs are hardest (21–88% recall); chained facts are solved at 100%. Prompt injection inverts rankings — chunk-level scorers lead, while Noveum peaks on out-of-scope fact pressure.
01. Code APIs hardest
Invented SDK methods slip through most often. Noveum lands at 75%.
Noveum recall75%
02. Chained facts solved
Every general-purpose judge hits 100% on chained-fact synthesis.
General judges100%
03. Pressure facts
Noveum posts perfect recall on out-of-scope fact pressure.
Out-of-scope recall100%
04. Injection inverts
Chunk-level scorers lead; Noveum's weakest substantial family at 55%.
Noveum recall55%
Section 07 –Relevance
Retrieval and answers
RAG evaluation splits into context-relevance (retrieval quality) and answer-relevance (did the response address the question). Ragas leads context at r = 0.95; Noveum separates answer-relevance at r = 0.78 while peers cluster near zero.
Context relevance is the field's strongest dimension and it is not a Noveum win. Ragas leads at r = 0.95 with Noveum essentially tied at 0.945 (held-out validated at r = 0.93 on 280 turns its scorer never saw in development). Nearly half of all turns retrieved nothing, and the strong scorers all treat an empty retrieval as a true zero.
Answer relevance separated the field sharply. Noveum correlates at r = 0.78 while every other native scorer lands between 0.04 and 0.27. One caveat: this dimension had low variance in the gold labels, so it carries less statistical power than the others.
Section 08 –Tool calls
Right tool, real arguments
Agent failures are wrong tool choice or fabricated arguments in otherwise plausible calls. Only four platforms ship native tool-call scorers. Noveum and DeepEval tie on selection; Arize balances fabricated-argument detection best.
01. Tool selection
Noveum and DeepEval tie at perfect specificity on tool choice.
Specificity100%
02. Fabricated arguments
Arize posts the best balance; Noveum is the most cautious autonomous gate.
Arize specificity81%
03. No universal profile
Gates want low false alarms; regression suites want exact-match recall.
Use-case split2 modes
Section 09 –Conversational
Staying in character
Role-adherence asks whether the agent held persona across multi-turn pressure. Only Noveum and DeepEval ship native scorers. Noveum leads at r = 0.79; GPT-tier judges cluster at 0.56–0.68.
Noveum's native conversational scorer leads at r = 0.79 and separates good agents from bad ones most cleanly. The GPT-tier scorers cluster at r = 0.56 to 0.68. On this task, the judge model dominates the plumbing unless the scorer is genuinely specialized.
Section 10 –Latency
Judge speed
For realtime gates and high-volume scoring, a slow judge is a non-starter regardless of accuracy. Noveum 0.59s, 2 to 27 times faster than the field.
Latency here is the local judge-model call, measured identically for every platform, not a hosted-API round-trip. A 13 to 16 second judge (Ragas, Patronus) cannot sit in a request path.
Accuracy and speed together leave exactly one point on the frontier: Noveum is more accurate and faster than every other platform. Coverage is the multiplier everyone forgets. Scoring six dimensions costs six scorer-calls per turn, so a calibrated, cheap judge compounds.
Section 11 –Consistency
High and flat
A platform can win one dimension and still be a poor general-purpose evaluator if it is spiky. Normalized scores across five dimensions. Low spread means balanced performance.
NA means the platform ships a dedicated scorer that runs hosted or server-side (excluded, not zeroed). A 0.00 cell is the worst measured value or a confirmed capability gap. The five middle columns are each platform's normalized scores per dimension. The final σ column is the spread of those five scores, not a score itself. Noveum's σ of 0.002 means its five dimension scores are all near 1.0 with almost no gap between them. Low σ at a high mean is reliable everywhere, unlike flat-at-the-bottom Patronus. Noveum's overall score is 0.999, never 0.002.
| Faith F1 | Ans-rel | Ctx-rel | Tool-sel | Role-adh | σ | |
|---|---|---|---|---|---|---|
| Noveum | 1.00 | 1.00 | 0.99 | 1.00 | 1.00 | 0.002 |
| DeepEval | 0.72 | 0.05 | 0.97 | 1.00 | 0.52 | 0.348 |
| Arize / Phoenix | 0.87 | 0.08 | 0.91 | 0.89 | 0.13 | 0.385 |
| Galileo | 0.92 | 0.00 | 0.83 | NA | 0.48 | 0.362 |
| Ragas | 0.49 | 0.35 | 1.00 | 0.13 | 0.48 | 0.287 |
| Maxim | 0.89 | 0.05 | 0.00 | NA | 0.44 | 0.358 |
| Braintrust | 0.90 | 0.31 | 0.16 | 0.00 | 0.00 | 0.333 |
| Patronus | 0.00 | 0.00 | 0.00 | 0.00 | 0.39 | 0.157 |
0.999
Noveum score
The headline number. Use this for any bar or chart.
0.002
Noveum spread (σ)
Consistency only. Lower is better. Never use as the score.
Noveum is the only platform both high and flat across every measured dimension, with a mean of 0.999 and a spread of just 0.002. Every other competitive platform carries at least one tall peak and one hole: DeepEval's near-zero answer relevance, Braintrust's missing tool and conversational scorers, Galileo's and Maxim's faithfulness over-flagging. If one platform must cover a whole agent, consistency is the number to check.
Section 12 –FAQ
Frequently asked questions
Reproducible · Open data
Download the full report
22-page benchmark plus buyer's guide. Every score reproducible.