NovaPilot

Analyze a project dataset, inspect ranked recommendations, and schedule recurring quality reports.

NovaPilot turns evaluation scores in a project dataset into a health report and a ranked list of recommendations to investigate. Run an analysis manually when you need an answer now, or create a scheduled report for recurring analysis.

NovaPilot does not run automatically after every evaluation. You launch it from Run Analysis or configure a Scheduled report.

NovaPilot is recommendations-only and does not require repository access. It does not edit or deploy your system prompt, model, parameters, tools, mapper, scorer configuration, or application code, and it never opens a pull request. Copy an accepted recommendation into your normal change process, verify the supporting evidence, and evaluate fresh traffic before rollout.

What you need

Before running NovaPilot, associate a dataset with the project and populate it with evaluated items. At least one item must have a valid scorer result: a finite, nonnegative score without a scorer error. Sparse score coverage may produce an Insufficient Data report; a dataset with no valid scorer results fails and must be evaluated first.

An analysis consumes 8 credits for each analyzed item with a valid scorer result, with a minimum of 800 credits per run, and supports at most 10,000 scored items per run. The run dialog shows the full filter-matching cohort, then estimates credits from the matching items that have evaluation scores. NovaPilot validates those results before analysis; sparse valid coverage can still produce an Insufficient Data report.

NovaPilot pages

Page	Use it for
Recommendations	Start an analysis, switch between recent reports, and review ranked recommendations
Chat with NovaPilot	View the current Coming Soon state; interactive chat is not available yet
Reports	Review the complete report history and open a report for export or email
Scheduled reports	Create, pause, edit, and inspect recurring dataset analyses

Opening Project → NovaPilot takes you to Recommendations.

Typical workflow

Associate a dataset with the project and run the evaluations whose scores you want NovaPilot to inspect.

Open NovaPilot → Recommendations and click Run Analysis.

Select the dataset, optionally filter its items, check the live item and credit estimate, and start the analysis.

Wait for the report to move from Queued to Processing and then Completed.

Review the health summary, scorer performance, failure patterns, system-prompt issues, and actionable recommendations.

If your team accepts a recommendation, implement it through your normal change process outside Noveum, collect and evaluate new data, then run NovaPilot again to compare the result.

Completed NovaPilot analysis showing the overall health grade, analyzed-item and score totals, the scorer-performance overview, and the beginning of Failure Patterns — Start with the overall health and evaluation coverage, then continue through the evidence sections before acting on a suggested change.

Manual and scheduled reports

A manual run and a scheduled run produce the same kind of NovaPilot report. Reports can contain:

an overall score and grade;
the number of items and scores analyzed;
scorer coverage and failure rates;
recurring failure patterns;
system-prompt findings;
recommendations ranked by priority, confidence, and implementation effort.

Open a completed report to download it as PDF or Markdown. From the full report page, you can also email it to yourself or the whole organization.

Availability

NovaPilot is enabled by default on the current Free through Enterprise plans, and each analysis consumes organization credits. An organization-specific entitlement override can still disable access. If NovaPilot is disabled for the active organization, the dashboard prompts you to contact support. If the credit estimate exceeds the remaining balance, the run dialog links to Settings → Usage & Credits.