Integrate noveum-trace

noveum-trace records application behavior as traces and spans. Initialization configures where spans are sent. Context managers and framework callbacks decide what application behavior is captured.

Install and initialize

Install only the tracing package in an application that already has its framework and provider dependencies:

pip install noveum-trace

import os

import noveum_trace

noveum_trace.init(
    api_key=os.environ["NOVEUM_API_KEY"],
    project=os.environ["NOVEUM_PROJECT"],
    environment=os.getenv("NOVEUM_ENVIRONMENT", "production"),
    service_version=os.environ["NOVEUM_SERVICE_VERSION"],
)

Initialization is process-wide. Configure it once before instrumented work begins. A later init() call does not reconfigure an active client.

Version agent behavior

NOVEUM_SERVICE_VERSION is your application or agent release identifier, not the noveum-trace package version.

Use a stable value for one behavior, such as support-agent-v3 or a release commit. Change it when an evaluation-relevant input changes:

• system or developer prompts
• model or model parameters
• tool definitions or routing logic
• retrieval and reranking strategy
• agent graph, handoffs, or stopping behavior

Noveum attaches service_version to traces, carries it into ETL items, and exposes it as a filter for comparing behavior, quality, latency, and cost between releases.

Integrate with a coding agent

Use this prompt in the repository that contains the application:

Integrate the Python package noveum-trace into this application without changing its runtime behavior. Initialize it once from NOVEUM_API_KEY, NOVEUM_PROJECT, NOVEUM_ENVIRONMENT, and NOVEUM_SERVICE_VERSION. Instrument the end-to-end agent operation and its LLM, retrieval, and tool operations. Preserve system prompts, user inputs, model outputs, token usage, available tool schemas, tool arguments and results, retrieval queries and ordered context, session or request identifiers, errors, and the terminal agent response. Flush buffered spans when a short-lived process exits. After integration, run representative successful and failed requests and report which required fields are present in each trace.

The Noveum Agent Skill can inspect the repository and guide the same workflow through the MCP server.

Capture the evidence evaluation needs

Capture broadly at instrumentation time. ETL can discard, combine, rename, and normalize fields later. Missing production evidence cannot be reconstructed after a request has finished.

Simple agents

Preserve these values for every model decision:

Tool-calling agents

Capture the complete decision boundary, not only the selected tool:

• available tool names, descriptions, argument schemas, and return schemas
• selected tool name and call identifier
• arguments passed to the tool
• result, success state, and error details
• model response after the tool result is returned

Use span.set_input_attributes(messages=messages, tools_available=tools) on an LLM span when those values are available. Record each tool execution on a child span:

from noveum_trace import trace_operation

with trace_operation("tool.lookup_order") as span:
    span.set_attributes({
        "function.type": "tool_call",
        "tool.name": "lookup_order",
        "tool.arguments": {"order_id": order_id},
    })
    result = lookup_order(order_id)
    span.set_attribute("tool.result", result)

An ETL mapper can normalize these spans into tools_available, tool_calls, parameters_passed, and tool_call_results.

Retrieval and RAG agents

Record retrieval separately from generation so scorers can connect an answer to the evidence that supported it:

from noveum_trace import trace_operation

with trace_operation("retrieval.search") as span:
    documents = search(query)
    span.set_attributes({
        "function.type": "retrieval",
        "retrieval.query": query,
        "retrieval.documents": documents,
    })

Keep context chunks ordered and retain document identifiers when available. The mapper converts them into retrieval_query and retrieved_context. Faithfulness and contextual scorers cannot run correctly when only a generated answer is captured.

Multi-step agents

Capture the complete parent-child structure, ordered turns, agent role and task, handoffs, terminal response, errors, and exit status. The mapper sets agent_exit on the final normalized item after it determines that the complete execution is present. agent_exit is not a generic raw span attribute that every instrumentation call must emit.

Record status and errors

Manual spans use SpanStatus values:

from noveum_trace import trace_operation
from noveum_trace.core.span import SpanStatus

with trace_operation("agent.plan") as span:
    try:
        plan = build_plan(task)
        span.set_attribute("agent.plan", plan)
        span.set_status(SpanStatus.OK)
    except Exception as exc:
        span.record_exception(exc)
        span.set_status(SpanStatus.ERROR, str(exc))
        raise

Do not pass strings such as "ok" or "error" to set_status().

Choose the integration surface

Verify completeness

Run a representative set of requests before configuring ETL:

1. one successful simple request
2. one provider or application error
3. one tool call, including its result
4. one retrieval request, including returned context
5. one complete multi-turn or multi-step execution

Inspect the spans and confirm that each field in the relevant capture table is present. Then continue with the evaluation workflow.