Run noveum-trace in production

Run noveum-trace at scale: configure release, user, session, payload, sampling, delivery, and shutdown behavior for a production Python service.

Use this guide after a synthetic trace has passed the quick setup. Production instrumentation should make incidents explainable without collecting more data than your organization permits.

Older material may call this operating model Noveum Scale. That is a legacy discovery term, not another product: use noveum-trace production configuration for sampling and delivery, and the Noveum MCP server for agent-accessible operational workflows.

The isolated package harness checks the noveum-trace==1.5.21 init signature and a fake-transport lifecycle. Configuration fences below are fragments to adapt and load-test; they are not universal production values.

The current Python SDK uses context managers and framework callbacks. It does not expose Noveum tracing decorators. Old examples that use a Noveum @trace or @span decorator do not apply to the current SDK.

Initialize once with stable dimensions

Initialize during process startup. Use a stable project and environment, and identify the deployed artifact with an immutable release, image digest, or commit SHA.

Code

import os
import noveum_trace

noveum_trace.init(
    api_key=os.environ["NOVEUM_API_KEY"],
    project="returns-agent",
    environment=os.environ.get("NOVEUM_ENVIRONMENT", "production"),
    service_version=os.environ["APP_RELEASE"],
)

You can also set the release with NOVEUM_SERVICE_VERSION. Do not use a moving label such as latest; version comparisons are useful only when the value identifies one deployed build.

Add filterable request context

Set trace metadata after the root operation starts. Use pseudonymous identifiers that remain stable only for as long as the investigation requires.

Code

from noveum_trace import get_current_trace, trace_operation

def handle_request(request):
    with trace_operation("support.request") as span:
        trace = get_current_trace()
        if trace is not None:
            trace.set_metadata(
                user_id=request.pseudonymous_user_id,
                session_id=request.conversation_id,
                request_id=request.request_id,
            )

        span.set_attribute("app.feature", "returns-assistant")
        return run_agent(request)

Trace metadata supplies the platform's user, session, and request dimensions. Custom span attributes remain useful when inspecting an operation, but a lookalike attribute such as session.id does not replace the trace's metadata.session_id field.

Capture inputs deliberately

capture_response() can extract supported response metadata and usage. It cannot infer the prompt or messages passed into the provider, and it does not record completion text. If an evaluation needs payload content, record an approved representation explicitly:

Code

with trace_llm_call(model="gpt-4o-mini", provider="openai") as span:
    safe_messages = build_policy_approved_messages(request)
    span.set_input_attributes(messages=safe_messages)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=safe_messages,
    )
    output_text = response.choices[0].message.content or ""
    span.set_output_attributes(response=output_text)
    span.capture_response(response)

Prefer an allowlist: omit credentials, authorization headers, cookies, raw tool secrets, unrestricted retrieved documents, and fields the evaluation does not need. Test the emitted trace, not only the application response, before rollout.

Add pseudonymization as defense in depth

The SDK can pseudonymize recognized PII patterns immediately before HTTP delivery. Use a deployment-specific salt from your secret manager:

Code

noveum_trace.init(
    project="returns-agent",
    environment="production",
    service_version=os.environ["APP_RELEASE"],
    security_config={
        "pii_enabled": True,
        "pii_salt": os.environ["NOVEUM_PII_SALT"],
    },
)

Pseudonymization is not a complete data-loss-prevention system: pattern matching can miss sensitive values. Minimize capture first, then treat pseudonymization as a second layer. Keep the salt stable only where cross-trace correlation is required, rotate it under your security policy, and never commit it.

SDK development payload files are local diagnostic artifacts and can contain the raw payload. Do not enable development payload capture with sensitive data, and protect or delete those files under your normal incident-data policy.

Sample traffic intentionally

At high volume, set a trace sample rate between 0.0 and 1.0. Start with a value your capacity plan supports and validate that important workflows remain represented.

Code

noveum_trace.init(
    project="returns-agent",
    tracing_config={"sample_rate": 0.1},
)

Random sampling can hide rare failures. If error retention is mandatory, enforce that requirement in your application or ingestion architecture rather than assuming a uniform sample preserves every error.

Tune delivery under load

The SDK buffers traces and retries deliveries. The following values are examples, not universal defaults; load-test them with the service's request rate and shutdown budget.

Code

noveum_trace.init(
    project="returns-agent",
    transport_config={
        "batch_size": 50,
        "batch_timeout": 2.0,
        "max_queue_size": 2000,
        "retry_attempts": 3,
        "retry_backoff": 1.0,
    },
)

Monitor application logs for authentication, queue, timeout, and delivery failures. Keep TLS verification enabled; use a trusted custom CA bundle for an approved corporate proxy instead of disabling certificate validation.

When the bounded queue is full, the SDK logs the condition and drops the affected trace. Size max_queue_size for measured bursts, alert on queue-full messages, and reduce capture or sampling before increasing memory without a limit. Do not rely on a client span-count setting to control an unbounded workflow; define intentional span boundaries and cap loops in the application.

For prefork servers, initialize noveum-trace after the worker process starts so each worker owns its exporter thread and queue. Do not initialize in the parent and then fork an already-running exporter.

Flush during shutdown

Long-running services export in the background. Once the service stops accepting new work, flush buffered telemetry before the process exits:

Code

def shutdown():
    stop_accepting_requests()
    wait_for_in_flight_requests()
    noveum_trace.flush()

Short workers and command-line jobs should put flush() in a finally block. Size the orchestrator's termination grace period so the application and exporter can finish.

Production checklist

A synthetic canary produces one trace with the expected project, environment, and release.
The trace's user, session, and request metadata use approved pseudonymous values.
Required evaluation inputs are present; prohibited payloads and credentials are absent.
Model, provider, token, and cost fields use the SDK's canonical llm.* names.
Sampling and buffer settings survive a representative load test.
A forced provider failure appears as an error when it escapes the context manager.
Graceful shutdown delivers a final short-lived trace.
Access, retention, deletion, and salt-rotation owners have approved the rollout.

Run noveum-trace in production

Initialize once with stable dimensions

Add filterable request context

Capture inputs deliberately

Add pseudonymization as defense in depth

Sample traffic intentionally

Tune delivery under load

Flush during shutdown

Production checklist

Next steps

On this page