NovaPilot

NovaPilot analyzes a selected scored dataset. It can summarize health, group recurring failures, review system prompts, and propose changes linked to supporting traces or items.

Analysis input

Choose one dataset plus optional item IDs or filters. NovaPilot reads existing NovaEval results; it does not automatically consume every Eval Job run.

Output

Depending on available evidence, a report can include:

• overall score, grade, and issue counts
• lower and higher performing scorers
• recurring failure patterns
• system prompt analysis
• prioritized recommendations with affected evidence
• PDF, Markdown, and email exports

Treat recommendations as hypotheses. Apply a change, rerun a comparable evaluation set, and compare results before release.

Scheduled analysis

Schedules rerun NovaPilot for a selected scored dataset on a cron expression and timezone. They do not trigger Eval Jobs or create new scorer results.

Start here

• NovaPilot workflow
• Run Eval Jobs
• Test changes with NovaSynth