Noveum documentation

Learn the complete noveum-trace, NovaEval, NovaPilot, NovaSynth, and NovaGuard (beta) workflow, from production evidence to a verified release.

Noveum is a closed-loop reliability platform for production AI agents. noveum-trace captures real behavior, NovaEval measures it, NovaPilot recommends what to investigate, NovaSynth exercises candidate changes, and NovaGuard adds runtime policies for enabled beta workspaces.

What you can do

The Noveum workflow

The products can be adopted independently. In the complete loop, NovaPilot stops at recommendations; your team implements and reviews the candidate change, NovaSynth tests it, and noveum-trace and NovaEval verify the behavior after deployment.

Send your first trace

Noveum's first-party SDK supports Python 3.9 and later.

pip install noveum-trace

Set your API key in the environment instead of placing it in source code:

export NOVEUM_API_KEY="your-api-key"

Initialize the SDK once, then wrap the operation you want to observe:

import os

import noveum_trace
from noveum_trace import trace_llm_call
from openai import OpenAI

noveum_trace.init(
    api_key=os.environ["NOVEUM_API_KEY"],
    project="support-agent",
    environment="development",
)

client = OpenAI()
messages = [{"role": "user", "content": "Where is my order?"}]

with trace_llm_call(model="gpt-4o-mini", provider="openai") as span:
    span.set_input_attributes(messages=messages)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
    )
    output_text = response.choices[0].message.content or ""
    span.set_output_attributes(response=output_text)
    span.capture_response(response)

noveum_trace.flush()

Open the project's Traces page after the script exits. The SDK creates the named project and environment when the first trace arrives.

From traces to evaluations

Tracing tells you what happened. Evaluation tells you whether the behavior was good enough.

1. Inspect representative traffic. Confirm that traces contain the inputs, outputs, IDs, and context you need.
2. Create an ETL job. Map relevant trace data into the StandardData schema at the granularity your evaluation needs.
3. Review and publish the dataset. Check field quality before using it as an evaluation source.
4. Create an eval job. Attach the scorers that match the fields and risks in your dataset.
5. Read scores with their explanations. Investigate failed items and missing-field results instead of relying on an average alone.
6. Run NovaPilot or NovaSynth when needed. Use NovaPilot for cross-item failure patterns and recommendations, and NovaSynth for controlled scenario testing.
7. Implement and verify the change. Your team owns the code or configuration change, review, deployment, and before/after comparison.

Core concepts

• A trace is one end-to-end application run.
• A span is one operation inside a trace.
• An attribute is searchable metadata attached to a trace or span.
• An event is a timestamped occurrence inside a span.
• A dataset item is the normalized unit an eval job scores.
• A scorer is one quality, safety, performance, or formatting check.

Framework integrations

Use the base SDK for custom Python applications, or start with a framework guide:

• LangChain
• LangGraph
• CrewAI
• LiveKit
• Pipecat

For direct workspace access from an MCP-compatible AI client, use the Noveum MCP server.

Get help

• Noveum dashboard
• noveum-trace issues
• Discord community
• Support