NovaEval
Evaluate stored application output in the Noveum platform or generate and score output with the NovaEval Python framework.
NovaEval measures AI quality, safety, retrieval, agent behavior, conversation behavior, voice signals, and application-specific criteria. The Noveum platform and the Python framework share scorer concepts but run different pipelines.
Noveum platform Eval Jobs
The platform evaluates output already stored in a dataset:
Dataset items -> Selected scorers -> Stored resultsAn Eval Job does not call the application model to generate a new answer. A model configured on an LLM-based scorer is the judge model, not the system being evaluated.
Use the platform when a team needs shared datasets, repeatable scorer configuration, realtime or manual runs, per-item results, and NovaPilot analysis.
Prepare compatible items
Create a dataset directly, through ETL, or through NovaSynth's managed pipeline.
Select scorers
Choose scorers whose inputs are present in the item. Recommendations provide a starting point but do not replace validation.
Run and inspect
Review scorer aggregates, item-level scores, pass verdicts, errors, and optional reasoning.
Diagnose the dataset
Open NovaPilot, select the scored dataset and useful failure filters, then generate an analysis.
Continue with Running evaluations.
NovaEval Python framework
The Python framework can generate model output as part of an experiment:
Dataset -> Target model or models -> Scorers -> Local resultsUse it for local experiments, model comparisons, CI, or custom infrastructure.
pip install novaevalfrom novaeval import Evaluator
from novaeval.datasets import MMLUDataset
from novaeval.models import OpenAIModel
from novaeval.scorers import AccuracyScorer
dataset = MMLUDataset(
subset="elementary_mathematics",
num_samples=10,
split="test",
)
model = OpenAIModel(
model_name="gpt-4o-mini",
temperature=0.0,
max_tokens=100,
)
evaluator = Evaluator(
dataset=dataset,
models=[model],
scorers=[AccuracyScorer(extract_answer=True)],
output_dir="./results",
)
results = evaluator.run()See novaeval on PyPI for the published framework.
Evaluation areas
| Area | Example question |
|---|---|
| Accuracy | Does the prediction match the reference answer or label? |
| RAG | Is the answer relevant and supported by retrieved context? |
| Conversation | Does the response use prior context and fulfill the user's intent? |
| Agent behavior | Did the agent select the expected tool and make progress? |
| Safety | Does the output violate policy or contain harmful content? |
| Voice and latency | Do the item media and metrics satisfy a compatible scorer contract? |
| Custom criteria | Does the output satisfy a product-specific rubric? |
Use the scorer guide to match evaluation goals with populated fields.