Benchmarking LLM-evaluation platforms

Today we're publishing a benchmark of eight LLM-evaluation platforms — Noveum, Arize/Phoenix, Braintrust, Maxim, Galileo, Patronus, DeepEval, and Ragas. We built a retrieval-augmented support agent, ran it through 100 adversarial conversations — 440 assistant turns, of which 131 were hallucinations — and asked each platform to find the agent's mistakes using its own native scorers, run reference-free in its documented best setup. Ground truth was set by a model that none of the platforms use: Claude Opus-4.8.

Across six dimensions, Noveum has the highest overall accuracy and the most even coverage, with an equal-weighted composite score of 0.999, against 0.650 for the next platform. The field is closer on individual dimensions, and several are led by other platforms — Ragas on context-relevance, Braintrust on false-alarm rate, and DeepEval on tool selection. All results are reported below.

Noveum builds one of the eight platforms, and the agent under test answers questions about Noveum's own documentation — a home-domain advantage that the neutral reference and the best-native competitor configurations reduce but do not fully remove. Ground truth came from a third party's model, every competitor ran in its strongest documented configuration, and the agent, the traces, the reference labels, every scorer, and the raw per-platform scores are all published, so any number here can be reproduced.

Results across every dimension

The table below summarizes the full study. Noveum leads or ties seven of the nine metrics; where another platform leads, the table records it: Braintrust has the lowest false-alarm rate (the share of correct answers a scorer wrongly flags) at 8%, Ragas the highest context-relevance correlation at 0.95, and DeepEval ties Noveum on tool selection at 100%.

Key Findings from the LLM Evaluation Platform Benchmark

-The benchmark found meaningful variation across evaluation dimensions, with different platforms leading different metrics rather than a single platform dominating every category.

-The equal-weighted composite score placed Noveum first overall, while Braintrust recorded the lowest faithfulness false-alarm rate, Ragas achieved the highest context-relevance correlation, and DeepEval tied for tool-selection specificity. Arize Phoenix consistently ranked among the top-performing platforms across multiple benchmark dimensions.

-Judge latency ranged from 0.59 seconds to 15.73 seconds, illustrating the trade-offs between evaluation quality and production throughput.

-The benchmark also evaluated operational workflows beyond scoring accuracy. Noveum was the only platform that automatically transformed raw production traces into ready-to-score evaluation items without manual field mapping.

-All benchmark artifacts—including the dataset, reference labels, scorer configurations, and raw platform outputs—are published, enabling every result to be independently reproduced, validated, and extended.

It summarizes the table; the per-dimension charts that follow show the detail.

Faithfulness

Faithfulness is whether each claim in an answer is supported by the retrieved context. It is the most important dimension for evaluation that runs without a person checking the output. A reliable scorer must combine high recall (it catches real hallucinations) with a low false-alarm rate (it rarely flags a correct answer).

Key Findings from the Faithfulness Evaluation

-Across 440 evaluation turns, Noveum detected 89% of hallucinations while incorrectly flagging 10% of correct answers, resulting in a Faithfulness F1 score of 0.84.

-Braintrust (80% recall / 8% false-alarm rate) and Arize (82% / 11%) closely followed. Their 95% confidence intervals overlap with Noveum's, making the top three platforms a statistical tie on recall and false-alarm rate.

-Ragas achieved the highest hallucination recall (94%) but also recorded the highest false-alarm rate (37%), making it less suitable as an automatic evaluation gate at that calibration.

-Patronus, a purpose-built hallucination model, detected 48% of hallucinations. It reliably identified blatant fabrications but missed the fluent, contextually plausible hallucinations that dominated this benchmark.

To check that this reflects the method and not simply a stronger judge model, we ran Noveum's scorer on the same GPT model the competitors use. It still scored higher than Braintrust's scorer there — 82% versus 55% recall — so the difference is in the method, not only the model.

One further factor shapes the faithfulness result: which of a platform's own scorers you choose. Braintrust's ClosedQA scorer flags 8% of correct answers; its autoevals.ragas scorer flags 66% on the same data. DeepEval's GEval catches 73% of hallucinations; its FaithfulnessMetric catches 1%. Both are real, documented scorers. The unit of comparison is therefore not the platform but the specific scorer, in a specific configuration, on a given kind of traffic.

The aggregate recall also hides how differently scorers handle different kinds of mistakes.

Multi-step reasoning errors were caught by every general-purpose scorer at 100%; only the narrow Patronus model missed them, at 41%. Fabricated code — a non-existent SDK method written inside otherwise-valid code — was the hardest case: recall ranged from 21% to 88%. Noveum caught 75% of fabricated code and 100% of pressure-to-answer fabrications; its weakest family was prompt injection, at 55%, where per-chunk scorers did better. Ragas's 88% on fabricated code reflects the same low precision that produces its 37% false-alarm rate elsewhere. Part of what makes this hard is that 78% of the agent's hallucinations came with no retrieval at all — the agent skipped the lookup and answered anyway — so a faithfulness scorer has to treat an empty retrieval as a real signal rather than abstain.

Relevance

Relevance has two parts: whether the retrieved chunks were relevant to the question (context-relevance), and whether the answer addressed the question (answer-relevance). We measure each as the correlation between a platform's score and the reference.

Key Findings from the Relevance Evaluation

-Relevance was evaluated across two complementary dimensions: context relevance, measuring how well retrieved documents matched the query, and answer relevance, measuring how well the final response addressed the user's question.

-Ragas achieved the highest context-relevance correlation (0.95), closely followed by Noveum (0.945) and DeepEval (0.92), indicating strong agreement with the reference on retrieval quality.

-Noveum achieved the highest answer-relevance correlation (0.78), while the remaining platforms ranged between 0.04 and 0.27, reflecting a much larger separation on this benchmark.

-The benchmark highlights that strong retrieval scoring does not necessarily translate into strong answer-quality evaluation. Performance varied substantially between context relevance and answer relevance across platforms.

-This is the lowest-powered dimension in the study — the reference's answer-relevance labels vary little on this dataset — so this should be read as the other scorers failing to separate good answers from bad ones here while Noveum's does, rather than as Noveum being several times better.

Tool calls

Agents also call tools, and the failures that matter are calling the wrong tool or calling the right one with a fabricated argument. Four platforms expose a tool scorer that runs locally; Braintrust and Patronus ship none, and Galileo and Maxim run theirs server-side. The chart shows specificity — how often a scorer correctly leaves a correct tool call alone.

Key Findings from the Tool Call Evaluation

-Noveum and DeepEval both achieved 100% tool-selection specificity, correctly identifying the appropriate tool for every evaluated interaction.

-Tool-parameter evaluation revealed a clear trade-off between strictness and practicality. DeepEval and Ragas caught every fabricated argument but flagged 65% of correct tool calls, resulting in a specificity of 35%.

-Arize provided the most balanced tool-parameter performance, achieving 81% specificity.

-Noveum prioritized leaving correct tool calls untouched, achieving the highest tool-parameter specificity (84%) while maintaining strong overall tool-selection performance.

-The appropriate evaluator depends on the deployment goal: strict exact-match validation for regression testing or higher specificity for automated production evaluation.

Here, the focus is on benchmark performance however, broader platform capabilities, engineering workflows, and deployment considerations should also be discussed to select the best AI Evaluation Platform for your agent.

Conversational behavior

Agents can also drift over the course of a conversation — abandoning their role, or conceding under pressure. Only Noveum and DeepEval ship a dedicated multi-turn scorer for this; the rest fall back to a general judge.

Key Findings from the Conversational Behavior Evaluation

-Noveum achieved the highest agreement with the reference (r = 0.79), followed by DeepEval (0.68). The remaining platforms ranged between 0.56 and 0.67.

-Only Noveum and DeepEval provide dedicated multi-turn role-adherence evaluators. The remaining platforms rely on general-purpose LLM judges for this task.

-Across all platforms, conversational role adherence proved more challenging than single-turn evaluation, with no platform achieving perfect agreement with the reference.

-Every platform—including Noveum—was more lenient toward the failing agent than the neutral reference, highlighting an important consideration when using conversational behavior scores as automatic deployment gates.

Speed

For scoring production traffic, speed matters alongside accuracy.

Key Findings from the Latency Evaluation

-Noveum recorded the lowest median judge latency (0.59 seconds) in the benchmark using gemini-3.1-flash-lite, followed by Arize (1.04s), Braintrust (3.30s), DeepEval (3.86s), Ragas (13.62s), and Patronus (15.73s).

-On this benchmark, no platform that achieved higher evaluation accuracy also recorded a lower judge latency, highlighting the balance between scoring quality and production performance.

-Latency measurements represent local judge-model-call time, measured consistently across all platforms. They do not include hosted API round-trip latency, ensuring a like-for-like comparison.

Trace-to-Evaluation Pipelines: The Noveum Difference

One practical difference comes before any score is computed: turning raw traces into clean, per-turn evaluation items. Noveum was the only platform that did this with no manual field-mapping — it read the trace structure and produced 440 ready-to-score items on its own. The other hosted platforms ingest traces but require the operator to map the fields; DeepEval and Ragas are libraries and do not ingest traces at all. For anyone evaluating platforms beyond benchmark performance, read the broader AI agent evaluation platform comparison.

The agent, the dataset, the reference labels, every scorer, and the raw per-platform scores are published, so each figure in this report can be re-derived and re-run on a different judge model, a different domain, or an additional platform.

If you're building an evaluation pipeline for production AI systems Book a call with Aditi to see how Noveum's evaluation pipeline differs. You can also join the early-access program to run these scorers on your own production traces.