Run Eval Jobs

An Eval Job applies configured scorers to selected items in one dataset. It stores a result per item and scorer, then aggregates those results for comparison and diagnosis.

Configure an AI provider

Scorers that call an LLM require an enabled organization provider with a name and model. Open organization provider settings, add the provider, enter its credentials, select a model, and test the connection.

Provider credentials are encrypted at rest. The provider form does not currently import a JSON configuration.

Create an Eval Job

Open Eval Jobs and create a job with:

• a descriptive name and optional description
• one dataset and dataset type
• one or more scorer configurations
• optional default item filters
• an enabled state for realtime evaluation of newly inserted items

The job identifies the dataset by ID and slug. It does not currently pin a published dataset release.

Select compatible scorers

Inspect representative item JSON before adding a scorer. A scorer can only evaluate evidence present in the item.

Recommend Scorers samples five items and combines that sample with full-dataset field statistics. Treat its output as a proposed starting set, then verify requirements and run a small test.

Built-in scorer configuration currently exposes the pass threshold in the job UI. The threshold determines the stored pass verdict for a numeric score.

Run the job

Manual runs can:

• select specific item IDs
• apply dataset item filters
• skip items that already have results
• force reevaluation
• temporarily override scorer selections
• use a configured sampling policy

Preview filters before a large run. A manual run does not have to evaluate every dataset item.

Realtime evaluation is separate from manual execution. When the job is enabled and linked to an eligible ingestion flow, newly inserted items can be queued automatically.

Inspect results

The result view is organized by scorer. Each scorer summary can include:

• total evaluations
• average, minimum, and maximum score
• p75, p90, p95, and p99 score
• passed and error counts
• average execution time

Expand a scorer to inspect its item results. Each result includes the item ID, numeric score, pass verdict, optional error, execution time, source trace ID when available, and optional reasoning.

Reasoning is not guaranteed. Its presence depends on the scorer and successful execution.

When results are absent or errors repeat:

1. inspect the dataset item in the JSON tab
2. confirm the scorer's required inputs are populated
3. check provider and model configuration for LLM-based scorers
4. review the result error
5. correct the mapper or scorer selection and run a small reevaluation

Analyze with NovaPilot

NovaPilot analyzes scored dataset items, not a specific Eval Job run. Open NovaPilot, select the dataset, optionally filter to failures or item IDs, and start an analysis.

A NovaPilot schedule runs recurring analysis over a selected scored dataset. It does not trigger an Eval Job. Ensure the dataset receives fresh scores before scheduled analysis runs.

Next steps

• Match scorers to fields
• Inspect dataset contracts
• Generate NovaPilot analysis