Noveum docs
ProductsNovaSynthRun tests and review resultsRun synthetic tests

Run tests and review results

Build persona-scenario test cases, launch a NovaSynth batch, inspect individual results, and share a report.

The Runs page is the central place to create and review NovaSynth batches. Each test case is one explicit persona-scenario pair and becomes one individual run.

Before you start

You need:

  • a completed NovaSynth agent setup;
  • at least one persona and one scenario;
  • an active LiveKit, Phone, or HTTP Chat connection;
  • organization credits and NovaSynth access;
  • scorers from NovaSynth's linked eval job if you want pass rates and evaluation analytics.

The scorer list comes from the eval job associated with the NovaSynth agent configuration. If the list is empty, you can still exercise the agent, but scored results and pass-rate charts will not be available.

Create a batch

Click New Test Batch on NovaSynth → Runs to open the four-step wizard. Next stays disabled until the current step is valid, and Launch run on the last step creates and starts the batch.

Name & notes: enter a Batch name, optional Notes, and an optional Test type such as regression or smoke. You can type a new test type to create your own tag.
Test cases: choose a persona, select one or more scenarios, and click Add combination. Repeat for every persona you need. Each persona-scenario pair becomes one run.
Scorers: select the scorers that run for this batch, or pick a preset from Select preset. Star a scorer with Mark as critical to make it count toward the batch's pass rates.
Connection: select an active connection, then set Concurrency and, for voice runs, Max call duration (s).
Click Launch run.

The run mode follows the agent type saved in Agent configuration. A chat agent runs in Text mode and the wizard lists only HTTP Chat connections. A voice agent runs in Voice mode and the wizard lists LiveKit and Phone connections. An HTTP Chat connection always runs in Text mode, and an inbound Phone connection always runs in Voice mode.

A batch supports up to 1,000 runs. The Test cases step shows how many runs remain. Start with a small set; increasing concurrency also increases simultaneous runs and provider demand.

The server checks available organization credits before it creates the batch. A text conversation is estimated at 12 credits per pair. For voice, the per-pair estimate is ceil((maximum duration in seconds / 60) × 100) credits. For example, a 90-second limit is ceil(1.5 × 100) = 150 credits per pair. This is a quota estimate based on the duration cap, not a promise that every call will use the full time.

Run options

OptionAvailable valuesEffect
Batch nameRequired, up to 200 charactersIdentifies the batch in Runs, reports, and comparison views
NotesOptional textRecords the purpose, release, or expected coverage
Test typeOptional tagLabels the batch on the Runs page
Test casesUp to 1,000 persona-scenario pairsCreates one individual run for every pair
ScorersScorers linked to NovaSynth's eval jobChooses which scorers evaluate each run
Mark as criticalAny selected scorerChooses which scorers decide pass rates
ConnectionActive project connections that match the agent typeChooses the endpoint every pair will exercise
Concurrency1 to 20, default 1Limits how many individual runs execute at once
Max call duration (s)60 to 1800 seconds, default 300Stops an individual voice run at the selected cap; voice mode only

Choose scorers

Only the scorers you select run for the batch. Critical scorers decide the outcome:

  • A run passes when every critical scorer passes.
  • When no scorer is marked critical, every selected scorer must pass.
  • Scorers you select but do not mark critical still appear on each run as Additional Scorers. They are tracked but do not affect the verdict.

Select preset offers Recommended, the product's suggested scorer set, and up to five scorer selections from previous runs. A previous-run preset also restores which scorers were critical. Marking more scorers as critical can lower the pass rate because every critical scorer must pass.

Inbound batches

Select an inbound Phone connection to test an outbound bot. After the batch is created, the Run IDs dialog lists every run ID with Copy all and Copy run id actions. Your dialler claims each run and calls its assigned Noveum number. To place one call by hand, click Get number on the run's row in the batch's All Runs table, then dial the number shown before its window closes. These batches use Voice mode; follow the inbound calls guide for the claim and dial workflow.

Batch lifecycle

Batch status can be Queued, Running, Completed, Partial Failure, Failed, or Cancelled. Individual runs additionally pass through Dispatching, and inbound runs can show Waiting for call or No call received.

While a batch is queued or running, Cancel batch in the Runs row menu and Cancel Batch on the batch detail page stop its queued and in-progress runs. A completed batch can be duplicated from the Runs row menu with Duplicate batch, and Rerun Test on the batch detail page does the same for any batch. Both open Duplicate Test Batch, which creates and immediately starts a new batch with the same personas, scenarios, and settings. Failed runs and inbound runs that received no call expose Retry run.

Search the Runs list by batch name, open Filters to narrow it by status or date range, or click Refresh. Each row shows the batch's pass rate, run count, success metrics, average duration, and average turns.

Analysis lifecycle

After a completed interaction, NovaSynth builds a dataset item, posts it to the dataset, and runs evaluation. A run's analysis shows Analysis is pending..., Building dataset item..., Posted to dataset., Evaluation in progress..., Analysis complete, or Analysis failed.

If the interaction completed without a synthetic trace, analysis fails because ETL cannot build the dataset item. Scorer or provider failures can also leave execution completed while analysis fails.

Inspect a batch

Open a row from Runs after it starts. The batch detail page contains:

  • a header with the batch's mode, agent, connection, and direction, plus Show details for the full configuration;
  • Total Runs, Execution Summary, Pass Rate, and Critical Scorers cards;
  • Scorer Breakdown, grouped by scorer category, with Only Critical and All views;
  • a NovaPilot Analysis card;
  • a persona-by-scenario Run Matrix;
  • an All Runs table.

Pass Rate counts the evaluated runs that passed all critical scorers. Critical Scorers shows how many of the batch's selected scorers are marked critical.

Completed Documentation Demo batch showing scorer breakdown, Run Matrix, and All Runs
A completed batch shows its scorer breakdown above the persona-by-scenario Run Matrix and the All Runs table. Open any completed entry to inspect its evidence.

Run Matrix

Each cell is one persona-scenario run, marked Passed, Failed, Error, or Evaluating, with the number of criteria met. Each row and the footer show the critical success rate. Use the toolbar to:

  • jump to a scenario or filter by scenario category;
  • filter by Status:;
  • sort by Critical success rate;
  • switch between Dense and Detailed cells.

Hover a cell to see its result and criteria met, then click it to open the run.

All Runs

All Runs groups runs by Scenario or Persona. Search by persona or scenario, filter by Passed, Failed, or Eval Error, and sort by duration, turns, or creation time. Expand a group and click View to open a run.

NovaPilot Analysis

Click Run NovaPilot Analysis on the card, or NovaPilot Analysis in the header, to analyze the batch with NovaPilot. The card shows the credits the analysis uses before you start. It needs an agent configuration, built dataset items, and scorer results. When the analysis completes, the card shows Overall Health, Traces Analyzed, Scores Evaluated, and Critical/High Issues, with a link to the suggested fixes in NovaPilot. If the analysis fails, click Retry.

Inspect one individual run

Open a run from the matrix or the All Runs table. The run page shows:

  • the persona and scenario, each with View details;
  • mode, turns or messages, duration, direction, and average score;
  • recording playback for runs with audio, as a Waveform or Simple audio player;
  • the transcript, with Download transcript;
  • Success Criteria, the critical scorers that decide the run's verdict, and Additional Scorers, which are informational;
  • the analysis summary and run summary;
  • Retry run when the run failed or received no call.
Confirm the status, persona, scenario, mode, and direction match the intended test case.
For voice, play the recording and compare it with the transcript. Use the turn buttons to jump between turns. For text, read the transcript in order.
Wait for analysis to complete, then review Success Criteria first. Filter by Failed or Passed, filter by Scorer type, sort by score, or click Expand All to read every scorer's reasoning.
If execution failed, read the recorded error and use Retry run only after correcting the likely connection, quota, or agent issue.
Completed Documentation Demo voice run showing waveform playback and ordered Assistant and User transcript
A completed voice run pairs waveform playback with the speaker-ordered transcript. Evaluation can still be processing after the conversation and recording are available.
NovaSynth scorer-result detail
Use the passed, failed, and scorer-type filters to investigate the exact criteria behind the outcome.

Do not infer a successful evaluation from a completed conversation alone. A run can finish the interaction while its dataset and scorer analysis are still processing.

Generate a report

Open the batch detail and click Generate Report in the header. While generation runs, the action shows Generating…; after completion it becomes View Report. If generation fails, click Generate Report again.

The Evaluation Report includes total runs, failures, average run duration, the all-criteria pass rate, average scores out of 10, and Individual Run Analysis. Each run's analysis lists its summary as bullets led by the recording times they cite, followed by a Test Setup block with the persona and scenario when the dataset carries simulation metadata. Voice runs also show conversation metrics, including STT word error rate when it was measured.

NovaSynth evaluation report
The report summarizes the completed batch and exposes Download Report and Email actions. Expand individual-run analysis before sharing when a headline metric needs context.

From the report dialog you can:

  • download PDF as novasynth-report-{batchRunId}.pdf;
  • download Markdown as novasynth-report-{batchRunId}.md;
  • email the report to yourself with Send to me;
  • email the report to the whole organization with Send to whole org.

Downloads use the interface language active when you generate them. For emailed reports, NovaSynth localizes the subject, body, and PDF for each recipient's stored language. The PDF cover identifies the batch, test mode, run totals, creation time, and organization so a downloaded or forwarded report retains its context.