Design Useful Traces

A trace should preserve one complete application execution, from the input that started it to the final result or error. If someone cannot reconstruct what the application saw, decided, called, and returned by opening one trace, the instrumentation is incomplete.

The trace model

The SDK creates identifiers, timestamps, duration, parent relationships, and lifecycle status. Your instrumentation supplies the application evidence in attributes.

Choose one complete trace boundary

Start one trace at the application boundary and close it only when that execution has reached a terminal result. Good trace boundaries include:

• one chatbot or voice-agent turn
• one complete RAG request
• one agent run across planning, tools, handoffs, and synthesis
• one independently evaluated item in a batch

Do not split one agent run into unrelated traces for retrieval, generation, and tools. ETL and evaluation need their shared parent-child structure to reconstruct the complete execution.

Use a stable, action-oriented root name such as support-agent.request, rag.answer-question, or document-agent.process. Keep user IDs, request IDs, and other high-cardinality values in metadata or attributes rather than span names.

Create spans at decision boundaries

Create a child span when an operation has its own input, output, latency, status, or evaluation meaning:

• each LLM decision
• each retrieval or reranking operation
• each tool or external API call
• each agent handoff
• each material routing or validation stage

Do not create a span for every helper function. Excess spans make traces harder to read without adding evidence.

Nested context managers preserve the hierarchy automatically:

from noveum_trace import trace_llm_call, trace_operation

with trace_operation("support-agent.request") as root_span:
    root_span.set_attributes({
        "agent.role": "customer_support",
        "agent.task": user_message,
    })

    with trace_operation("retrieval.search") as retrieval_span:
        documents = search(user_message)
        retrieval_span.set_attributes({
            "function.type": "retrieval",
            "retrieval.query": user_message,
            "retrieval.documents": documents,
        })

    messages = build_messages(user_message, documents)
    with trace_llm_call(
        model="gpt-4o-mini",
        provider="openai",
        operation="answer.generate",
    ) as llm_span:
        llm_span.set_input_attributes(messages=messages)
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
        )
        llm_span.capture_response(response)
        llm_span.set_output_attributes(response=response)

    answer = response.choices[0].message.content or ""
    root_span.set_attributes({
        "agent.response": answer,
        "exit_status": "completed",
    })

Treat attributes as the span payload

Timing shows where a request was slow. Attributes explain what happened and make evaluation possible. Capture the complete decision boundary, not only labels or summary metrics.

LLM spans

Preserve:

• system and developer instructions
• ordered input messages or prompt
• model output, including tool-call requests
• model and provider identity
• generation parameters that affect behavior
• input, output, and total token usage
• available tool names, descriptions, and schemas when tools were offered

Recommended keys include llm.system_prompt, llm.input.messages, llm.prompt, llm.output.response, llm.completion, llm.model, llm.provider, and the llm.usage.* token fields.

Tool spans

Record each tool execution as a child span with:

• function.type set to tool_call
• tool name and call identifier
• exact arguments
• result and success state
• exception type and message when it fails

The LLM span should also retain the complete tools offered to the model. ETL can then normalize the evidence into tools_available, tool_calls, parameters_passed, and tool_call_results.

Retrieval spans

Record retrieval independently from generation. Preserve the query, ordered returned chunks, document identifiers, similarity values when available, and reranking output. Recommended keys include function.type, retrieval.query, and retrieval.documents.

Faithfulness and contextual scorers cannot determine whether an answer is grounded when the trace contains only the generated response.

Agent and workflow spans

Preserve the role, task, ordered turns, decisions, handoffs, terminal response, errors, and exit status. agent_exit belongs to the normalized dataset item and should be set by the ETL mapper only after it confirms that the complete execution is present. It is not a required raw span attribute.

Use events sparingly

An event is useful when a timestamped occurrence matters inside one long-running span and does not deserve its own duration, such as a retry scheduled during a streaming operation.

Prefer:

• an attribute for the final state of a span
• a child span for work with its own duration or result
• the span exception and error status for failures

Avoid start and completion events that duplicate the span timestamps. Avoid events for model inputs, outputs, retrieval context, or tool results; those values belong in attributes on the relevant span.

Record status and errors

Context managers close spans when an exception is raised, but record the exception and explicit status when you handle errors inside the operation:

from noveum_trace import trace_operation
from noveum_trace.core.span import SpanStatus

with trace_operation("tool.lookup-order") as span:
    try:
        result = lookup_order(order_id)
        span.set_attribute("tool.result", result)
        span.set_status(SpanStatus.OK)
    except Exception as exc:
        span.record_exception(exc)
        span.set_status(SpanStatus.ERROR, str(exc))
        raise

Use SpanStatus.OK and SpanStatus.ERROR. Do not pass string values such as "ok" or "error" to SDK spans.

Add correlation and release context

At the trace boundary, attach identifiers needed to find related activity:

• user, session, conversation, and request identifiers
• environment and project
• tenant or account identifiers that are safe to transmit
• feature, route, or workflow labels
• service_version for the deployed agent behavior

Set NOVEUM_SERVICE_VERSION to a stable release identifier and change it when prompts, models, tools, retrieval, routing, or stopping behavior changes. This makes quality, cost, and latency comparable between releases.

Protect sensitive and oversized data

Never send API keys, authorization headers, credentials, or private keys. Define a redaction policy for personal or regulated data before capturing production prompts and outputs.

Keep span names and common filter attributes low-cardinality. Store identifiers and variable payloads in dedicated attributes. Avoid binary data and unbounded objects; retain the ordered text, IDs, schemas, and results needed to debug and evaluate the operation.

Verify before configuring ETL

Inspect representative traces and confirm that:

1. the root span covers the complete execution
2. child spans preserve the actual operation order and parent relationships
3. every LLM decision includes its input, output, model, provider, and usage
4. tool spans include offered schemas, selected calls, arguments, results, and errors
5. retrieval spans include queries and ordered context
6. handled failures include exception details and error status
7. the terminal response and exit status are present
8. the trace carries the expected environment, request identifiers, and service version

Continue with the noveum-trace integration guide, the Python SDK reference, or REST trace ingestion. After representative traces are complete, configure the evaluation pipeline.