NovaSynth
Exercise voice and chat agents with synthetic personas and scenarios before production issues reach users.
NovaSynth connects a synthetic user to a reachable agent. An LLM follows the selected persona and scenario, while Noveum tracks execution, builds an evaluation item when trace processing succeeds, and runs the project's linked scorer configuration.
Core concepts
| Resource | Purpose |
|---|---|
| Agent Config | Describes the agent under test and provisions its managed evaluation resources |
| Persona | Defines the synthetic user's behavior, language, speaking style, and optional profile context |
| Scenario | Defines actions and conditions the synthetic user should attempt during the interaction |
| Test Connection | Stores one supported route to the deployed agent |
| Run | Executes one persona and scenario against one connection |
| Batch | Executes selected persona and scenario pairs with controlled concurrency |
| Analysis | Tracks dataset creation and scorer execution after the interaction finishes |
Supported connections
| Connection | Voice mode | Text mode | Configuration |
|---|---|---|---|
| LiveKit | Supported | Supported | LiveKit URL, API key, API secret, and agent worker name |
| Phone | Supported | Not applicable | Destination number and country calling code |
| HTTP Chat | Not applicable | Supported | HTTP endpoint, authentication, request shape, and response extraction |
Direct Pipecat, WebSocket, VAPI, Retell, and ElevenLabs Conversational endpoints are not currently executable by the NovaSynth worker. A Pipecat-backed agent can be tested through a supported phone number or compatible HTTP chat interface.
Execution and analysis are separate
A run can complete its interaction while analysis is still pending or has failed. Dataset insertion and scoring depend on a synthetic trace, successful ETL, a linked enabled Eval Job, and compatible scorer inputs.
Production noveum-trace instrumentation is not required to execute a NovaSynth session. Add production tracing when you also need observability for real traffic or application-specific evidence beyond the managed synthetic trace.
Pass-rate definitions
Batch success criteria select which scorer results determine the headline pass rates. They do not change the full scorer list configured on the linked Eval Job.
- Pass rate: share of analyzed sessions where every selected success criterion passed
- 70% threshold: share of analyzed sessions where at least 70% of selected success criteria passed
Only sessions with completed analysis and usable scorer results enter these calculations. When no explicit success criteria are selected, result views can fall back to available scorer results.
When to use NovaSynth
| Goal | Approach |
|---|---|
| Pre-release quality gate | Run a repeatable set of key personas and scenarios |
| Regression test | Duplicate a batch against changed agent behavior and compare results |
| Edge-case coverage | Add scenarios for failures found in production or support reports |
| Language coverage | Create personas and scenarios for each supported language |
| Deployment comparison | Run separate batches against separate connections |
NovaSynth has no built-in nightly run scheduler. Use external automation with the public API when synthetic batches must run on a schedule.
Get started
Configure the agent
Complete NovaSynth setup with a non-empty system prompt and a supported connection.
Create test behavior
Add personas and scenarios manually or generate drafts asynchronously.
Run a small batch
Start with one persona-scenario pair and verify connectivity, execution, trace processing, and scoring.
Scale and compare
Create a batch run, select success criteria, and inspect the matrix, individual sessions, reports, and analytics.
API users can use the generated OpenAPI reference for current request and response contracts.