Noveum documentation
Capture production behavior, prepare evaluation data, measure quality, and validate improvements.
Noveum supports two improvement loops. Instrument a production application with noveum-trace when you need evidence from real traffic, or start with NovaSynth and an existing dataset when you want to test a single agent before adding production tracing.
Choose your starting point
Production evidence
Start with noveum-trace
Capture real requests, map them into evaluation-ready items, measure failures, and verify the next release.
noveum-trace -> traces -> ETL -> dataset -> NovaEval -> NovaPilot -> NovaSynth
Send your first tracePre-production testing
Start without production tracing
Use NovaSynth conversations or an existing dataset to evaluate a single agent, prioritize changes, and test again.
NovaSynth or dataset -> NovaEval -> NovaPilot -> implement -> NovaSynth
Set up a synthetic testBoth paths converge on a versioned dataset and the same evaluation workflow. You can add noveum-trace later without rebuilding the rest of the loop.
Production trace workflow
Capture
Install noveum-trace, initialize it once, and instrument the model, retrieval, tool, and agent operations that contain evaluation evidence.
Inspect
Confirm that representative traces include model inputs and outputs, system prompts, token usage, tool schemas and results, retrieval queries and context, and stable request identifiers.
Prepare
Use an ETL job to map each trace into the fields required by your dataset. The mapper decides whether one trace produces zero, one, or many items.
Evaluate
Run NovaEval with scorers that match the fields your mapper produced.
Pick an integration
| Application | Start here | What it covers |
|---|---|---|
| Python or a custom LLM app | SDK integration | Initialization, capture fields, and verification |
| LangChain | LangChain integration | Chains, tools, agents, and retrieval callbacks |
| LangGraph | LangGraph integration | Nodes, transitions, loops, and parent relationships |
| LiveKit | LiveKit integration | Conversations, STT, TTS, LLM, VAD, and tool activity |
| Pipecat | Pipecat integration | Voice pipelines, processors, and transport activity |
| CrewAI | CrewAI integration | Crews, tasks, tools, and handoffs |
| Any language | REST trace ingestion | Direct single-trace and batch ingestion |
| Coding agents | Agent skill | Guided integration and trace-completeness review |
Product guides
NovaEval
Measure quality, safety, retrieval, tool use, and task outcomes.
NovaPilot
Group failures, identify likely causes, and prioritize engineering changes.
NovaSynth
Exercise agents with repeatable personas and scenarios before release.
Reference
If you are new to Noveum, continue with the five-minute quick setup.