NovaPilot

NovaPilot analyzes a selected dataset after NovaEval results exist. It combines item content, scorer results, failures, and optional scorer reasoning into a report with health, patterns, and recommended changes.

NovaPilot does not automatically receive every completed Eval Job run. Start an analysis explicitly, or schedule recurring analysis for a scored dataset.

Prepare the dataset

Before running NovaPilot:

1. populate a dataset through ETL, NovaSynth, or direct item ingestion
2. run compatible NovaEval scorers
3. confirm enough representative items have score results
4. choose useful item or failure filters

NovaPilot requires scored items. It cannot diagnose quality from unscored dataset content alone.

Run an analysis

Open Project > NovaPilot > Recommendations and choose Run Analysis.

Select:

• one dataset
• optional specific item IDs
• optional dataset filters, including failure-oriented filters

Preview the filter count when narrowing a large dataset, then start the analysis. The request identifies a project and dataset, not an Eval Job or Eval Job run.

Read the report

Report sections depend on the available data.

Overall health

The health card can show:

• overall score and grade
• health summary
• traces analyzed
• scores evaluated
• total issues and critical or high-priority issues
• dataset and generation time
• insufficient-data warning when the sample is too small

Scorer performance

The scorer section groups lower and higher performing scorers. It can include score summaries, counts, and failure rates. It does not currently provide per-scorer historical trend charts or a built-in worst-item sample for every scorer.

Failure patterns

Patterns group recurring issues and connect them to affected scorers, traces, or items when evidence is available. Use those links to inspect the underlying dataset content before accepting a recommendation.

System prompt analysis

When a system prompt is present, the report can identify prompt issues and propose changes. Treat generated text as a candidate patch. Test it against the same regression set before release.

Recommendations

Recommendations can include priority, category, affected evidence, confidence, current state, suggested change, expected impact, and scorer-specific improvements. Availability varies by analysis result.

Export and share

A completed report can be:

• downloaded as PDF
• downloaded as Markdown
• emailed to the requester or configured team recipients

Schedule recurring analysis

A NovaPilot schedule stores:

• name
• dataset
• cron expression
• timezone
• enabled state
• email recipients

The schedule runs NovaPilot analysis over already-scored dataset items. It does not trigger an Eval Job, change scorers, or create fresh scores.

Ensure the selected dataset receives new items and scores before each scheduled analysis. Schedule runs can be pending, processing, completed, or failed, with run history available from the schedule detail.

Dashboard sections

Improvement loop

1. inspect a recommendation and its evidence
2. change the prompt, tool, retrieval, or agent behavior
3. test the candidate with NovaSynth or a controlled environment
4. produce new dataset items and NovaEval scores
5. rerun NovaPilot against a comparable item set
6. release under a new service_version and monitor production traces

Continue with Run Eval Jobs, NovaSynth runs, or the evaluation workflow.