The closed-loop workflow

Connect noveum-trace, NovaEval, NovaPilot, NovaSynth, and NovaGuard beta in an evidence-driven workflow from production behavior to a verified release.

Noveum's five products cover different stages of one AI reliability loop. Start by completing one end-to-end path. Use the product guides when you need deeper configuration.

You can use each product independently. The complete loop is most useful when every aggregate result can be traced to a stable dataset item and a recorded execution.

The workflow

NovaPilot stops at recommendations. It does not edit application code, change a prompt, or open a pull request. Your team owns implementation, review, testing, deployment, and rollback.

1. Capture one production-like execution

Instrument one complete request, conversation, or workflow with Quick SDK setup. Record the model, tool, retrieval, agent, or voice operations needed to explain the outcome.

Before moving on, verify:

• the trace is in the intended project and environment;
• parent and child spans match execution order;
• status and duration are plausible;
• stable session_id and service_version values are present when available;
• no secrets or unnecessary personal data were captured.

Use Project Traces when the project is known or Explorer to browse across the organization.

2. Convert representative evidence into a dataset

An ETL job selects project traces and runs a saved Python mapper. The mapper can exclude a trace, create one conversation-level item, or create several turn or operation items.

The trace remains source evidence. The versioned dataset becomes the stable evaluation surface. See Create an ETL job, Build and test the mapper, and ETL run filters.

3. Measure with NovaEval

An eval job binds a dataset to saved scorer configuration. Choose the item population, confirm scorer compatibility and thresholds, review the item and credit estimate, then start the run.

Before trusting an aggregate score:

1. check item and scorer coverage;
2. separate failed verdicts from scorer execution errors;
3. open one result and its dataset item;
4. follow the source trace link;
5. confirm the result describes application behavior rather than missing mapper input or a scorer mismatch.

Use Set up, run, and review an eval job and Choose scorers. For conversation or step sequences, open Runs.

4. Prioritize an investigation with NovaPilot

Run NovaPilot only after the dataset and scorer results are trustworthy. A report can summarize performance, recurring failure patterns, system-prompt findings, and ranked recommendations.

Treat each recommendation as a hypothesis supported by the available evidence. Open affected items and source traces, then decide whether the appropriate change belongs in the application, prompt, model, parameters, retrieval, tool, mapper, or scorer.

Copy an accepted recommendation into your normal engineering process. NovaPilot does not apply it. Continue with NovaPilot reports and Recommendations.

5. Test the candidate with NovaSynth

NovaSynth calls the agent connection you configure and records controlled voice or text conversations. Build a test matrix from reusable personas and scenarios that represent happy paths, edge cases, user behavior, accents, interruptions, or adversarial requests relevant to the change.

Start with one persona and scenario pair:

1. verify the connection and transcript;
2. confirm the linked eval job and the scorers marked critical for the batch;
3. inspect scorer coverage and errors;
4. expand to the release test matrix only after the first run is valid;
5. compare the candidate with the behavior you intended to improve.

NovaSynth does not change the connected agent. See Set up NovaSynth, Create personas, Create scenarios, Build a test matrix, and Review runs.

6. Release and close the loop

After your normal review and deployment process:

1. emit a new service_version with the release;
2. compare the same NovaEval checks before and after the change;
3. filter noveum-trace to the old and new production cohorts;
4. inspect unexpected differences at the span level;
5. run NovaPilot again only after enough new evaluated evidence exists;
6. keep or revert the change based on production and synthetic evidence.

A recommendation or passing synthetic run is not the final outcome. The loop closes when the intended production behavior improves without unacceptable regressions.

Where NovaGuard fits

NovaGuard is the runtime policy product and is currently beta. The documented controls cover enabled cost caps and request or token rate limits for connected traffic. These policies sit around live provider traffic; they do not replace tracing, evaluation, investigation, or pre-release testing.

See NovaGuard beta for the currently available controls.

Diagnose the right layer

For a complete synthetic example, use the Demo workspace. For your own application, begin with Quick SDK setup.