Choose an Evaluation Workflow
Build an evaluation loop from production traces, synthetic conversations, or an existing dataset.
Every Noveum evaluation runs against dataset items. What changes is how those items are created.
Path 1: Learn from production traffic
Use this path when real requests are the evidence you want to improve.
Instrument the application
Add noveum-trace and instrument the operations that carry model, retrieval, tool, and agent behavior. Initialization configures delivery; it does not automatically capture application code that has not been instrumented.
Check representative traces
Inspect successful, failed, simple, tool-using, and retrieval-heavy requests. Verify the required inputs, outputs, context, identifiers, and usage fields before creating a mapper.
Map traces into a dataset
Create an ETL job. Its mapper can emit zero, one, or many items from each trace. Choose the item boundary that matches the behavior you want to score, such as one complete conversation or one item per turn.
Evaluate and diagnose
Run NovaEval. Then open NovaPilot, select the scored dataset and useful failure filters, and generate an analysis.
Test and release
Exercise the candidate behavior with NovaSynth, release it under a new service_version, and compare the new production traces with the previous version.
Path 2: Start without production tracing
Use this path for a single agent that can be exercised before production instrumentation is available.
Create evaluation data
Configure NovaSynth personas, scenarios, and a supported connection. Initial setup creates or reuses a linked dataset, ETL job, and Eval Job for synthetic sessions. You can also import an existing dataset or add normalized items directly.
Run NovaEval
Select scorers whose required fields are present in each item. Use scorer recommendations when you need help choosing a starting set.
Apply NovaPilot recommendations
Review repeated failure patterns, update the prompt or agent logic, and preserve the original dataset as a regression set.
Test again
Repeat the same NovaSynth scenarios and rerun NovaEval. Add noveum-trace later when production behavior becomes part of the acceptance criteria.
Capture contract for the trace-first path
Scorers consume normalized dataset fields, so the source trace must retain enough evidence for ETL to produce them.
| Agent behavior | Preserve in the trace | Typical dataset fields |
|---|---|---|
| Simple model call | System prompt, user input, model output, model and provider, token usage | system_prompt, input_text, output_text |
| Tool calling | Available tool names and schemas, selected tool, arguments, result, and errors | tools_available, tool_calls, parameters_passed, tool_call_results |
| Retrieval or RAG | Retrieval query, ordered context chunks, generated answer, and optional ground truth | retrieval_query, retrieved_context, output_text, ground_truth |
| Multi-step agent | Agent role and task, ordered turns, terminal response, exit status, and completion signal | agent_role, agent_task, conversation_context, agent_response, exit_status, agent_exit |
The agent_exit field belongs to the mapped dataset item. Set it on the final agent item when the mapper has the complete execution. It is not required on every raw span.
What the mapper controls
The ETL mapper defines the evaluation unit and the target schema. It can:
- discard health checks, incomplete requests, and irrelevant spans
- combine model, retrieval, and tool spans into one item
- emit one item per conversational turn or one item for the complete conversation
- normalize provider-specific attributes into the StandardData schema
- preserve
source_trace_id,service_version, session identifiers, and evidence needed for diagnosis
Review generated mapper code before activating it. A mapper cannot recover prompts, tool results, or retrieval context that the application never captured.
Run an evaluation
- Open the project and create an Eval Job.
- Select the dataset you want to measure.
- Choose compatible scorers or request scorer recommendations.
- Run the job and inspect per-item scores, pass or fail verdicts, and reasoning.
- Open NovaPilot, select the scored dataset and useful filters, then generate an analysis when enough representative results exist.