Analytics
Compare NovaSynth pass rates across batches, coverage, persona and scenario trends, and scorer failures.
Open NovaSynth → Analytics after runs and their linked eval-job scoring complete. Analytics compares batches over a date range and applies each batch's critical scorers to its pass-rate views.
Choose the date range
The range applies to Pass Rate Trend Across Recent Batches, Batch Comparison, the persona and scenario trend, the scorer trend, Top Failing Scorers, and the CSV export. Scenario Coverage and Persona Coverage always count every run in the project.
Pass rates
NovaSynth reports two pass rates for evaluated runs:
- Pass Rate counts a run when every critical scorer scores 7 or higher with no scorer error.
- 70%-Criteria Pass Rate counts a run when at least 70% of its critical scorers score 7 or higher and none is unavailable. Comparison and trend cards label this rate Critical Success Rate.
Critical scorers are the ones starred when the batch was created. When a batch has no critical scorers, every scorer it ran counts. Runs without completed scorer results are excluded from both rates. See Choose scorers.
Cards
| Card | What it answers |
|---|---|
| Pass Rate Trend Across Recent Batches | How did the pass rate and the 70% threshold rate move from batch to batch? Only batches with evaluated runs appear. Hover a batch to see its evaluated and total runs. |
| Batch Comparison | How do two to four chosen batches differ on Pass Rate, Total Runs, Personas, Scenarios, Criteria Met, Critical Success Rate, Avg Duration, and Total Duration? |
| Scenario Coverage and Persona Coverage | Which scenarios and personas are the most and least tested, with run counts and average duration? |
| Persona × Scenario Critical Success Rate Trend | How did one persona and scenario pairing perform across batches? |
| Scorer Critical Success Rate Trend | How often did one scorer pass in each batch that ran it? |
| Top Failing Scorers | Which critical scorers fail most often? Up to ten scorers appear, ordered by failure rate. |
Compare batches
Criteria Met counts passing scorer results out of all scorer results in the batch, so it shows how close a failing batch came to passing.
Follow a pairing or scorer
Persona × Scenario Critical Success Rate Trend starts on the first persona and scenario. Choose another Persona: and Scenario: to plot that pairing's critical success rate for each batch that evaluated it, with the criteria met for each point.
Scorer Critical Success Rate Trend starts on the first scorer. Choose another Scorer: to plot, for each batch that ran it, how many runs that scorer passed. Scorer errors count toward the batch total but not toward passes.
Coverage
The coverage cards list the top five scenarios and personas. Switch between Most tested and Least tested, or click View all to search the full list.
Read the page in order
- Validate the population. Check the date range and each batch's evaluated and total runs. A low evaluated count can make a pass-rate change look more significant than it is.
- Locate the change. Use Pass Rate Trend Across Recent Batches to find when quality moved, then compare the batches on either side in Batch Comparison.
- Check coverage. Use the coverage cards to confirm that both batches exercised comparable scenarios and personas.
- Isolate the behavior. Use the persona and scenario trend and the scorer trend to find which pairing or criterion changed.
- Find the failing behavior. Use Top Failing Scorers, then open the batch and individual transcripts for evidence.
Analytics is a comparison surface, not a substitute for run inspection. A scorer failure can have different causes across two transcripts even when it contributes to the same aggregate.
Empty cards
An empty card does not always mean there are no runs:
- Pass-rate cards require completed scorer results.
- The persona and scenario trend needs completed runs for the chosen pairing.
- Top Failing Scorers appears only when scorer pass or fail data is available.
- A shorter date range may exclude batches that are visible on the Runs page.
If a batch has runs but no evaluated runs, open an individual run and check its analysis status and linked eval-job scorer results. Then confirm that the batch marked the scorers you intend to aggregate as critical.
Export CSV
Click Export CSV to download novasynth-analytics.csv for the selected range. The file contains these sections:
- pass rate by date;
- run counts by scenario category;
- the five newest batches in the range with their pass rate, evaluated runs, and total runs;
- average duration and run count by endpoint;
- every batch in the range with its date, pass rate, 70%-criteria pass rate, evaluated runs, and total runs.
Column headings follow the current interface language. Export after choosing the intended date range. The CSV reflects that range and does not include the coverage cards.