Using scorers

Choose compatible published or custom scorers, understand their required fields and native scales, and interpret results correctly.

NovaEval's public directory currently contains 112 published scorers across 15 categories. The generated Scorer catalog is the authoritative list for current availability, category, cost class, description, and native score scale.

This guide explains how to choose and configure scorers. It intentionally does not duplicate the catalog's full table.

Before you start

Inspect representative item JSON and define the product behavior you need to measure. For model-judged checks, configure an AI provider if the scorer uses organization credentials. Use a one-item evaluation to validate unfamiliar inputs and thresholds before a large run.

Start with the evaluation question

Choose a small set of scorers that corresponds to the behavior you need to improve:

• answer quality and task completion;
• retrieval relevance, grounding, and hallucination;
• conversation coherence and role adherence;
• tool selection and parameter correctness;
• safety, refusal, moderation, and bias;
• latency, telephony, and audio quality;
• reference-based text similarity when ground truth exists.

More scorers create more results and may consume more credits without making the decision clearer. Use scorer recommendations when you want suggestions derived from a dataset sample, then review compatibility before saving the eval job.

Required fields determine compatibility

Every scorer expects particular StandardData fields. Typical examples include:

A scorer cannot infer a field that the ETL mapper did not write. Preview representative mapper output, compare it with the scorer's requirements in the directory, and exclude incompatible scorers before running a large job.

Preserve each scorer's native scale

Published scorers normalize their primary numeric result into four catalog scale families:

• 0-10 scores with a default pass threshold of 7;
• 1-10 scores;
• audio-derived measurements normalized to 0-10;
• millisecond measurements transformed to a 0-10 score.

Some scorers also return boolean flags, labels, reasoning, or structured evidence as auxiliary output. Those values do not replace the scorer's published numeric scale. The Results and Health views aggregate values within a scorer, so do not compare raw averages across different scale families. Configure the pass threshold in the same scale as the scorer and use the generated catalog's Scale column as the source of truth.

Free and premium execution

The catalog identifies each published scorer as Free or Premium. Scorers labelled Free do not consume evaluation credits. Premium scorer charges depend on input size and scorer type; see billing rates. Model-backed scorers can also require configured provider access.

See Billing, credits, and quotas before scheduling a high-volume job and Smart Sampling when full-traffic evaluation is unnecessary.

Create a custom scorer

Open Organization → Scorers → New Custom Scorer. The editor provides:

• AI chat to propose a name, description, prompt template, pass threshold, placeholders, and judge settings from a plain-language evaluation requirement;
• a manual form for entering the same configuration directly.

Custom prompt templates reference StandardData values with {field} placeholders:

template: |
  User task: {agent_task}
  Agent response: {agent_response}
  Evaluate whether the response is complete, correct, and actionable.

The editor warns when a placeholder root is not in the StandardData catalog. Review that warning, the proposed 0-10 rubric, and the pass threshold before saving. Drafts are stored in the current browser while you work.

Edit a custom scorer

Open Organization → Scorers, find a custom scorer, and select Edit on its card. Published scorers are read-only, but for a custom scorer you can change:

• its name and description;
• the prompt template and its StandardData placeholders, which determine the fields supplied to the scorer;
• the pass threshold;
• the judge provider, model, and temperature.

Select Save changes to save the new definition. Existing result rows keep the score, pass/fail state, and reasoning recorded when they ran; saving an edit does not recompute historical results.

An eval job stores its own snapshot of the scorer configuration, including the prompt, judge settings, and threshold selected for that job. Editing the organization-level custom scorer does not refresh that snapshot. Before the next run, open each affected eval job, choose Edit Scorers, remove the custom scorer, and add it again to capture the new definition. Review the replacement configuration before saving. Completed runs and result rows remain unchanged.

Read results

An eval result stays linked to its scorer, dataset item, eval job, and run when that metadata is available. Use the aggregate view to find a weak scorer, then expand individual results to inspect score, pass/fail state, reasoning, execution time, error details, and source trace.

For the complete workflow, including run selection and result filtering, see Eval jobs.

Expected result

The eval job contains a small, intentional scorer set. Every scorer has the required mapped inputs, a reviewed threshold in its native scale, and a known Free or Premium classification. Results can be traced from an aggregate row to item reasoning and source evidence.

Troubleshooting

Next steps

• Scorer catalog
• StandardData schema
• Eval jobs