NovaSynth

Create synthetic users and scenarios, run them against a live agent connection, and inspect evaluation results.

NovaSynth runs synthetic voice or text conversations against an AI agent before real users encounter the same cases. You define who the synthetic user is, what situation they should exercise, and which deployed connection NovaSynth should call.

From setup to results

Each test case is one persona-scenario pair. A batch can contain many pairs, and each pair becomes an individual run against the selected agent connection.

Generate personas and scenarios with AI

NovaSynth can propose personas and scenarios from the context you provide. At a customer-facing level, the workflow is:

Save the agent's system prompt and optional testing instructions in Agent Configuration.

Open Personas or Scenarios and select Generate with AI.

Review the pre-filled prompt, choose how many drafts to create, and add instructions for the coverage you want.

Start generation and wait for the job banner to report completion.

Review and edit every generated draft before adding it to a run.

The system prompt supplies domain and behavior context. Additional instructions can focus the output on specific languages, user behaviors, happy paths, edge cases, or expected refusals. The generation dialog lets you adjust this context for one job without first changing the saved project configuration.

AI generation creates editable test inputs. It does not modify the deployed agent, change its source code, or start a conversation. A real test begins only when you explicitly start a run against a configured connection.

Available pages

Page	What you do there
Overview	Check setup progress, run totals, pass-rate trends, coverage, and recent batches
Agent Configuration	Maintain the agent details, agent type, system prompt, and testing instructions
Agent Connections	Register and manage LiveKit, Phone, or HTTP Chat endpoints
Personas	Create or generate synthetic users with different behavior and voice settings
Scenarios	Create or generate conversation flows and edge cases
Runs	Build persona-scenario pairs, launch batches, and inspect results
Analytics	Compare batches, coverage, persona and scenario trends, and failing scorers

NovaSynth opens on Overview.

Before the first batch, Overview guides you through four setup steps: Agent Config, Personas, Scenarios, and Connections. After those steps are complete, it offers Create your first run. Once batches exist, it shows total runs, personas, scenarios, and the configured agent, a banner for any running batch with Cancel run, the pass-rate trend, scenario and persona coverage, and the five most recent batches. The trend chart offers Last 7d, Last 30d, Last 90d, and All. Use Runs for paginated history and Analytics for comparisons. + New run in the page header opens Runs once setup is complete.

Pass-rate definitions

NovaSynth aggregates an individual run against the scorers marked critical for its batch:

Pass Rate, also called the all-criteria pass rate, is the percentage of evaluated runs where every critical scorer scored 7 or higher without a scorer error.
70%-Criteria Pass Rate, shown as Critical Success Rate in comparisons and trends, is the percentage of evaluated runs where at least 70% of the critical scorers scored 7 or higher.

Runs without completed scorer results do not contribute a pass rate, and neither do runs whose critical scorers are only unavailable or not applicable with none failing. If no scorer is marked critical, NovaSynth uses every scorer the batch ran. Only the scorers selected for the batch run; see Choose scorers.

Get to a first result

Open a project and select NovaSynth.

Save the agent configuration with the agent's system prompt.

Add an active agent connection.

Create at least one persona and one scenario.

Open Runs, click New Test Batch, name the batch, and add one persona-scenario combination.

Select scorers, star the critical ones, choose a connection, and click Launch run.

Open the individual run to inspect its transcript, recording when present, timing, and scorer results.

Start with one pair and concurrency 1. Once the connection and evaluation path work, expand to a larger matrix.

Supported connection modes

Connection	Run modes	Intended use
LiveKit	Voice in the dashboard; Voice or Text through the API	A LiveKit agent deployment
Phone	Voice only	Outbound calls to an agent, or inbound tests where your bot calls a Noveum number
HTTP Chat	Text only	A text agent exposed through an HTTP endpoint

In the dashboard, the run mode follows the Agent Type in Agent configuration: a Voice Agent runs in Voice mode and a Chatbot in Text mode. Voice runs support a per-call duration limit.

Test an outbound bot

For an agent that places calls, configure an inbound Phone connection. NovaSynth creates persona-scenario run IDs; your script claims a run and has the bot dial the assigned number. See Inbound calls.

Reports and analytics

After a batch has results, its detail page shows pass metrics, a scorer breakdown, a NovaPilot analysis card, a persona-by-scenario matrix, and the complete run list. Generate a batch report to view aggregate and individual-run results, then download it as PDF or Markdown or email it to yourself or the organization.

Analytics become useful after individual runs finish and their linked eval-job scorers complete. Use the 7-, 30-, or 90-day views, or all time, to compare batches and spot regressions.