Scenarios

Define the situations and conversation steps each synthetic user should exercise.

A scenario describes what should happen in a synthetic conversation. Reuse a scenario across different personas to see whether the same workflow succeeds for different user behavior, language, and voice conditions.

NovaSynth Scenarios page with Generate with AI and Create Scenario actions plus a plan-selection scenario showing its description, step count, and edit and delete controls — Use **Create Scenario** for a specific scripted situation or **Generate with AI** to propose editable coverage from the saved agent context.

Create a scenario manually

Open NovaSynth → Scenarios and click Create Scenario.

Enter a short name and a description of the behavior or outcome the test should exercise.

Click Add Step for each conditional user action, keeping the steps in conversation order.

Under Planned Interruptions, click Add Interruption when the caller should cut in at a planned point.

Add coverage tags and choose a Category, then click Create and review the scenario's description and step count in the list.

Create Scenario dialog showing the required name, description, empty Conversation Steps area with Add Step, and tags — The initial form starts with no conversation steps. Give the scenario a clear outcome, then click **Add Step** to describe the synthetic user's actions.

Field	Use
Name	Required, concise label used in the run matrix
Description	What the test is intended to exercise
Conversation Steps	Ordered conditional actions for the synthetic user
Planned Interruptions	Points where the caller cuts in while the agent is talking
Tags	Labels for organizing scenario coverage
Category	Groups the scenario on the Scenarios page and in the run wizard; type a new name to create a category

Click Add Step for each conversation event. A step has:

an optional Condition, such as agent asks for account number;
a required Action, the line the caller says, such as I'd like to dispute a charge.

Use the move controls to reorder steps.

Create Scenario dialog with one numbered Conversation Step containing an optional condition, the caller action, a remove control, and the Add Step button — Each numbered step pairs an optional observable condition with what the synthetic user should say. Add or remove steps until the expected conversation flow is unambiguous.

Keep conditions and actions explicit enough that the synthetic user can follow them from observable conversation events.

Plan interruptions

Each planned interruption makes the caller cut in while the agent is speaking. Choose a Type:

Group	Types	What the caller says
Fixed line	Ask to repeat, Hold on, Talk to someone else, Hurry up, Can't hear you, Not interested	A fixed line for that type, once the agent has been talking for Threshold (seconds)
Generated	Ask a question, Correct the bot, Answer early	A generated line that gets the Intent across

Then place the interruption:

Marker turn is the agent turn near which the caller starts watching for a moment to cut in.
Arm after event ties the interruption to a conversation step, so the caller cuts into whatever the agent says next after that step. When it is not set, NovaEval infers the moment from the description, which is less reliable.
Window (turns) is how many agent turns the caller keeps trying.
If missed chooses Force it on the next turn or Drop it when no good moment appears.

Interruptions are armed by marker turn, so an entry with a lower marker turn runs earlier even when it appears further down the list. NovaEval fires at most four interruptions per call and skips the rest.

Generate scenarios with AI

Generate with AI starts an asynchronous generation job using the configured agent prompt.

Open Scenarios and click Generate with AI.

Review the Agent System Prompt, pre-filled from Agent Configuration.

Enter the Number of Scenarios as a whole number of 1 or more, and optional Additional Instructions describing the coverage you want.

Click Generate and wait for the generation banner to complete.

Edit the resulting descriptions, steps, conditions, and tags before running them.

Generate Scenarios dialog pre-filled from the saved support-agent prompt, with scenario count, an empty additional instructions field, and the Generate action — The saved system prompt is copied into the dialog; Additional Instructions start empty. Adjust the count and instructions to request distinct test situations, then click **Generate** to create editable drafts.

After submission, the Scenarios page displays the active generation job:

NovaSynth Scenarios page with an AI is generating scenarios progress banner above a No scenarios yet empty state — The progress banner confirms that generation is asynchronous. This new project has no saved scenarios yet; review the generated descriptions, steps, and tags after the job completes.

Generation proposes reusable test records; it does not contact the connected agent or start a batch. Review each draft for unsupported assumptions, duplicate coverage, impossible prerequisites, and steps that depend on information a synthetic user cannot observe.

Good instructions ask for distinct outcomes rather than near-duplicate wording. For example:

Generate a successful cancellation, an invalid-account case, a retention-offer objection, and a request outside the agent's allowed scope.

Review and organize scenarios

The list shows each scenario's name, description, step count, and category. Search it with Search scenario..., filter by category, or select Group by category while the All filter is active. Click a row to see the full scenario, including its steps and planned interruptions, or use the row actions to edit or delete it. A scenario that existing runs use cannot be deleted.

To run scenarios, open Runs → New Test Batch and pair them with personas in the Test cases step. See Run tests and review results.

Design useful coverage

Include a clear happy path so connection failures are distinguishable from behavior failures.
Add one expected refusal or out-of-scope case.
Write observable steps instead of implementation details hidden from the synthetic user.
Keep each scenario focused on one outcome.
Use tags to separate regression, edge-case, language, or release-specific coverage.