NovaEval overview
Evaluate production AI systems in the Noveum platform or run the open-source NovaEval framework directly from Python.
NovaEval is Noveum's evaluation engine for measuring the quality and reliability of LLM applications, agents, RAG systems, and voice experiences. It supports a broad scorer library, custom evaluation logic, reusable datasets, and detailed result reporting.
Choose how you want to evaluate
Use the Noveum platform
Create Eval Jobs from dashboard datasets, compare per-item results, and send completed runs to NovaPilot.
Create an Eval JobUse the Python framework
Compose datasets, models, and scorers in code for local experiments, CI workflows, or custom infrastructure.
Open the NovaEval repositoryEvaluation model
NovaEval follows one consistent pipeline:
Dataset -> Model -> Scorers -> Results| Component | Responsibility |
|---|---|
| Dataset | Supplies inputs, expected outputs, context, and metadata |
| Model | Produces the output being evaluated |
| Scorer | Measures a specific property such as accuracy, relevance, safety, or latency |
| Results | Stores item-level scores, aggregate metrics, reasoning, and artifacts |
What NovaEval measures
| Evaluation area | Typical questions |
|---|---|
| Accuracy and classification | Did the model produce the expected answer or label? |
| RAG quality | Was the answer grounded, relevant, and supported by retrieved context? |
| Conversation quality | Did the response preserve context, follow the intended role, and complete the user's goal? |
| Agent behavior | Did the agent select the right tool, follow the expected path, and complete the task? |
| Voice quality | Was speech clear, correctly pronounced, and free from audio artifacts? |
| Latency | Were time to first token, STT, TTS, and end-to-end response times acceptable? |
| Safety | Did the output avoid harmful, biased, or policy-violating content? |
| Custom criteria | Did the output satisfy requirements unique to your product or domain? |
See the scorers reference for the available scorers and their input requirements.
Run an evaluation in the dashboard
Prepare a dataset
Create a dataset directly or use an ETL job to transform production traces.
Create an Eval Job
Select the dataset, evaluation mode, and scorers. Use scorer recommendations when you want NovaEval to inspect the dataset shape and propose a starting set.
Review results
Compare aggregate statistics, inspect individual failures, and read scorer reasoning for each item.
Improve the system
Send a completed run to NovaPilot to group recurring failures and prioritize fixes.
The full workflow is covered in Running evaluations.
Run NovaEval from Python
Install the open-source framework:
pip install novaevalCreate a small evaluation:
from novaeval import Evaluator
from novaeval.datasets import MMLUDataset
from novaeval.models import OpenAIModel
from novaeval.scorers import AccuracyScorer
dataset = MMLUDataset(
subset="elementary_mathematics",
num_samples=10,
split="test",
)
model = OpenAIModel(
model_name="gpt-4o-mini",
temperature=0.0,
max_tokens=100,
)
evaluator = Evaluator(
dataset=dataset,
models=[model],
scorers=[AccuracyScorer(extract_answer=True)],
output_dir="./results",
)
results = evaluator.run()The framework also supports configuration-driven evaluations, custom datasets, custom model adapters, custom scorers, and multiple result formats.
Platform and framework together
Use the dashboard when teams need shared datasets, repeatable Eval Jobs, visual result analysis, and NovaPilot recommendations. Use the Python framework when evaluation must run inside an existing codebase or CI system. Both approaches follow the same dataset, scorer, and result concepts, so teams can move between them without changing how they reason about quality.