Runs and Results
Launch individual or batch synthetic sessions and separate execution status from evaluation analysis.
A run executes one persona and scenario against one active connection. A batch groups up to 1,000 unique persona-scenario pairs under one endpoint, mode, and execution configuration.
Individual and batch runs
| Run type | Pair selection | Use case |
|---|---|---|
| Individual | One persona and one scenario | Connectivity check or focused debugging |
| Scenario launch | All combinations of selected personas and scenarios | Coverage matrix |
| Runs-page batch | Explicit unique pairs or full combinations | Curated regression set |
Batches can be launched from Scenarios or Runs. A batch uses exactly one endpoint. Run separate batches to compare deployments.
Configure a batch
| Setting | Contract |
|---|---|
| Name | Required batch identifier |
| Endpoint | One active LiveKit, phone, or HTTP Chat connection |
| Mode | voice or text, compatible with the endpoint |
| Pairs | Explicit pairs, or persona IDs crossed with scenario IDs |
| Concurrency | 1 through 20, default 1 |
| Maximum voice duration | 60 through 600 seconds, default 300 |
| Success criteria | Scorer IDs used for batch pass-rate calculations |
The cross product cannot exceed 1,000 sessions. Concurrency controls simultaneous execution; a new batch is sequential by default.
Success criteria do not choose which scorers execute. The linked Eval Job sends its configured scorer set. Selected criteria determine the headline pass rates and failure gate.
Execution statuses
Individual run
| Status | Meaning |
|---|---|
queued | Waiting for a worker |
dispatching | Claimed and executing the interaction |
running | Valid API state for an active interaction |
completed | Interaction completed; analysis can still be pending or failed |
failed | Dispatch or interaction failed |
cancelled | Execution was cancelled |
Batch
| Status | Meaning |
|---|---|
queued | Batch was created |
running | At least one session has started |
completed | Execution finished without run failures |
partial_failure | At least one run completed and at least one failed |
failed | Runs failed and none completed |
cancelled | All runs were cancelled |
Execution status does not represent scorer success. A run remains completed when the conversation succeeds but one or more criteria fail.
Analysis lifecycle
After a completed interaction, NovaSynth can move through item construction, dataset insertion, evaluation, and terminal analysis. The UI can show analysis as pending, building an item, running evaluation, completed, or failed.
If the interaction completed without a synthetic trace, analysis fails because ETL cannot build the dataset item. Scorer or provider failures can also leave execution completed while analysis fails.
Read batch results
The batch detail page combines:
- execution progress, including completed, failed, and cancelled runs
- pass rate and 70% threshold rate for analyzed sessions
- selected success-criteria breakdown
- persona by scenario matrix
- flat list of individual runs
The matrix distinguishes execution failure from criteria failure. A dash means a persona-scenario pair was not included.
Read an individual result
The run detail identifies the persona and scenario separately and can show:
- execution status, mode, endpoint, turns, and duration
- transcript when available
- audio player and media metrics when a recording exists
- aggregate numeric score when scorer results exist
- per-scorer score, pass verdict, error, and reasoning when supplied
Audio, transcript, metrics, and reasoning are conditional. Their absence does not by itself mean the interaction failed.
Reports and analytics
After eligible batch analyses become terminal, use Generate Report. When generation completes, View Report provides available PDF and Markdown downloads.
Analytics can aggregate pass rates, scenario distribution, average and total duration by persona, and failing scorers. CSV export contains aggregated analytics rather than raw run payloads.
Diagnose failures
- separate execution status from analysis status
- inspect endpoint and mode compatibility
- inspect the transcript or execution error
- confirm a synthetic trace ID exists
- inspect managed ETL and dataset insertion
- inspect linked Eval Job scorer results and provider errors
- duplicate a focused batch after applying the fix
Continue with NovaPilot when enough scored dataset items exist for pattern analysis.