Evaluate Your AI Agents Like a Pro
112 published scorers. Configured evaluation. Repeatable quality evidence for your agents.
Traditional metrics like accuracy are insufficient for AI agents. A multi-dimensional approach can measure accuracy, safety, cost-efficiency, and more. NovaEval connects reviewed datasets to compatible scorers so teams can evaluate agents repeatably.
112
Built-in Metrics
The Challenge
Why Evaluating AI Agents is Hard
Evaluating AI agents is fundamentally different from evaluating traditional software or ML models. Agents are non-deterministic, complex, and often produce novel outputs that don't match any pre-defined 'correct' answer.
Observability Alone is Useless
AI agents can take many steps per execution, producing more traces and spans than a team can review one by one. Configured evaluation helps narrow the review set, while linked trace evidence lets investigators inspect the operations behind a failed scorer decision.
Beyond Accuracy
Traditional accuracy is a poor measure for AI agents. An agent might produce an answer that's not in your training set but is still correct, or a technically accurate answer that's not helpful.
The Hallucination Problem
AI agents can confidently produce false information (hallucinations). You need specific metrics to detect and measure hallucinations, not just overall accuracy.
Safety & Compliance
You need to ensure agents don't produce toxic, biased, or non-compliant outputs. This requires specialized evaluation metrics for safety, bias detection, and regulatory compliance.
Cost-Quality Trade-offs
You need to balance quality with cost. A more expensive model might produce better results, but is the improvement worth the extra cost? You need metrics to measure this trade-off.
Eval Makes Observability Actionable
NovaEval stores per-item scorer results and links mapped items back to source traces. Model-backed scorers can include reasoning, and NovaPilot can organize recurring failures into evidence-backed patterns for review.
The NovaEval Solution
NovaEval: Comprehensive Agent Evaluation
NovaEval is Noveum.ai's evaluation engine for AI agents. Its library has 112 published scorers across 15 categories: 82 Premium, 30 Free, 76 model-backed, and 36 dedicated voice and audio scorers. Eval jobs apply only selected compatible checks to configured dataset items.
112 Published Scorers
Choose compatible scorers for the quality, safety, behavior, latency, and audio dimensions relevant to your agent
Agent Scorers
- Tool Relevancy Scorer
- Task Progression Scorer
- Goal Achievement Scorer
- Role Adherence Scorer
Conversational
- Coherence Scorer
- Empathy Scorer
- User Satisfaction Scorer
- Persona Adherence Scorer
RAG Quality
- Faithfulness Scorer
- Context Relevancy Scorer
- Answer Correctness Scorer
- Context Recall Scorer
Safety & Bias
- Toxicity Detection Scorer
- Bias Detection Scorer
- PII Detection Scorer
- Response Safety Scorer
Hallucination Detection
- Factual Accuracy Scorer
- Claim Verification Scorer
- Groundedness Scorer
- Context Faithfulness Scorer
Advanced Quality
- Information Density Scorer
- Clarity & Coherence Scorer
- Technical Accuracy Scorer
- Citation Quality Scorer
112 Published Scorers
Choose compatible published scorers for accuracy, safety, RAG performance, latency, audio, and other documented evaluation needs.
- Accuracy metrics: correctness, similarity, relevancy
- Safety metrics: toxicity, bias, PII detection
- RAG metrics: faithfulness, context recall, groundedness
Key Benefits
Why Teams Choose NovaEval
NovaEval provides comprehensive agent evaluation that goes beyond traditional testing.
Broad Evaluation Coverage
Measure configured quality, safety, compliance-related, cost, and performance criteria instead of relying on accuracy alone. Your team decides which evidence and thresholds are appropriate for the use case.
Faster Quality Gates
Use repeatable evaluation results as release evidence before deployment. Your team owns any approval rule, implementation change, and rollout decision.
Data-Driven Decision Making
Use evaluation data to inform model, prompt, parameter, and tool decisions. Move from assumptions to measured evidence while keeping release decisions with your team.
Continuous Improvement
Use selected production data to create evaluation datasets, identify recurring failures, and add focused test cases that can reveal later regressions.
Audit and Review Evidence
Document the datasets, configured checks, thresholds, and results used in an AI review. Evaluation reports can support an audit process, but they do not establish legal or regulatory compliance on their own.
Move from Guessing to Evidence
Use repeatable evaluation results as evidence for production-readiness decisions. Your team defines the datasets, scorers, thresholds, and release criteria.
Getting Started
How NovaEval Works
Get started with comprehensive agent evaluation in four simple steps.
- 1
Create an Evaluation Dataset
Use an ETL job to select traces and transform them with a reviewed mapper. Run a bounded batch first, then enable realtime processing when the mapper is stable.
- 2
Select Scorers
Choose from 112 published scorers, or use NovaEval's Scorer Recommender to suggest compatible checks from sampled dataset items.
- 3
Set Up Eval Jobs
Bind a populated dataset to compatible scorers, then run a selected cohort on demand or enable processing for eligible new ETL items.
- 4
Get Detailed Reports
Review NovaPilot reports that summarize issues in evaluated dataset items and recommend next steps for your team.
Evaluation Workflow
The NovaEval workflow is designed for both one-time evaluation and continuous monitoring. Here's what happens when you run an evaluation:
- 1An ETL mapper turns selected traces into dataset items
- 2Use the Scorer Recommender or select scorers directly
- 3An eval job scores the item cohort you select
- 4NovaPilot analyzes completed evidence and recommends next steps
Real-World Applications
How Teams Use NovaEval
See how organizations use NovaEval to measure configured quality criteria and collect release evidence.
Pre-Production Quality Gates
Evaluate new agent versions before deployment. Run the configured evaluation suite, review its results against team-defined thresholds, and use that evidence in the rollout decision.
Example: A fintech company requires 95% accuracy and 0% toxicity before any agent update goes live.
Model Selection
Compare already-captured outputs from candidate models on equivalent datasets while holding the scorer configuration fixed.
Example: A legal tech team captures outputs from five candidate models and scores each cohort with the same legal-reasoning rubric.
Prompt Optimization
Capture each prompt version separately, evaluate equivalent cohorts with the same scorer configuration, and compare the results.
Example: An e-commerce team captures equivalent cohorts for two prompt versions and compares the same satisfaction scorer.
Regression Detection
Compare repeatable evaluation results across releases and trace cohorts. Use service-version filters and saved scorer decisions to investigate a change before rollout.
Example: A healthcare platform evaluates a reviewed pre-release cohort, finds an accuracy regression, and keeps the candidate version from production.
Connected Workflow
Part of the Noveum.ai Ecosystem
NovaEval integrates tracing, evaluation, and NovaPilot analysis in one review workflow.
The Continuous Improvement Loop
Traces capture agent behavior, evaluation measures selected dataset items, and NovaPilot turns those results into recommendations. Your team reviews and implements changes, then evaluates again.
- 1
Trace
Capture behavior
- 2
Evaluate
Measure quality
- 3
NovaPilot
Review recommendations
- 4
Improve
Apply & repeat
Why NovaEval Stands Out
vs. Braintrust
Braintrust is evaluation-focused only.
NovaEval connects tracing and evaluation with NovaPilot analysis reports.
vs. DeepEval
DeepEval is an open-source library requiring self-management.
NovaEval is a managed platform with 112 published scorers and configurable evaluation pipelines.
vs. Custom Solutions
Building custom evaluation is time-consuming and error-prone.
NovaEval provides published scorers plus rubric-based custom scorers; each still requires compatible dataset fields and configuration.
Start Evaluating Your Agents Today
Build a reviewed dataset, choose compatible scorers, and inspect the resulting decisions and source evidence.
Explore more solutions