ETL Jobs

ETL Jobs automatically convert your raw trace spans into structured evaluation items, one per LLM call, so every agent decision can be scored independently.

What is an ETL Job?

An ETL Job is the bridge between your raw traces and your evaluation dataset.

Your application sends traces to Noveum — each trace is a full record of one request through your agent. An ETL Job reads those traces and extracts the individual LLM calls, turning each one into a structured dataset item that NovaEval can score.

Key concept: one item per LLM call.

Each LLM call inside a trace becomes one independently-evaluated dataset item. A chatbot conversation with 5 turns produces 5 items. A multi-step research agent with 8 LLM calls produces 8 items. This granularity lets you score every individual decision your agent made — not just the final result.

Once configured, an ETL Job runs continuously so your dataset stays fresh as new traces arrive.

flowchart LR
    Traces["Your traces\n(one per request)"] --> ETL["ETL Job\n(AI mapper)"]
    ETL --> Items["Dataset items\n(one per LLM call)"]
    Items --> Eval["Eval Job\n(NovaEval scorers)"]
    Eval --> NovaPilot["NovaPilot\n(recommendations)"]

Creating an ETL Job

Navigate to your project's ETL Jobs tab and click New ETL Job.

Step 1 — Name and project

Give the job a descriptive name (e.g., production-support-bot) and select which Project to pull traces from.

Step 2 — Target dataset

Choose the Dataset where items will be loaded. You can create a new dataset inline from this step.

Step 3 — Environment

Select the trace Environment (e.g., production, staging) to source traces from. Only environments where your SDK has sent traces will appear.

Step 4 — Generate the mapper

Click Generate Code in the Mapper tab. Noveum reads a sample of your recent traces and writes mapper code for you automatically.

The mapper code defines how each span in your trace gets turned into a dataset item. You review it in the editor, make any adjustments using Improve Code, then apply it.

See Using the AI Mapper for a full step-by-step guide to generating, reviewing, and improving mapper code.

The Mapper Editor

After generation, the mapper code appears in an editor. You can:

Review the generated code before running it
Edit manually if you need to adjust specific field mappings
Improve Code — describe a problem in plain English and the AI rewrites the relevant section
Run the mapper against your sample trace to preview the items it would produce

For a full walkthrough of generating, reviewing, and improving mapper code — including examples of what each generated section does and how to handle common problems — see Using the AI Mapper.

Status — running, completed, failed
Items produced — how many items were loaded into the dataset
Errors — if the mapper encounters a problem, the details are shown here
Duration — how long the run took

Fixing failures

If a run fails, click Fix with AI next to the error. Describe what went wrong in plain English and the AI updates the mapper code to address it. Then click Run again to verify.

Where ETL fits in the full workflow

flowchart TD
    SDK["Integrate SDK"] --> Traces["Traces flow to Noveum"]
    Traces --> ETL["ETL Job extracts items\n(one per LLM call)"]
    ETL --> Dataset["Dataset\n(versioned item store)"]
    Dataset --> Eval["Eval Job\n(attach scorers, run)"]
    Eval --> NovaPilot["NovaPilot\n(failure patterns + recommendations)"]

Tips

Start with a staging environment to validate your mapper before running against production traces.
If items look wrong, use Improve Code and describe the problem — it is faster than editing the code manually.
If you update your agent (e.g., add new fields or change the trace structure), click Generate Code again to refresh the mapper.

Next steps

Using the AI Mapper — step-by-step guide to generating and improving mapper code
What is a Dataset — understand the items the mapper produces
Running Evaluations — score your dataset items with NovaEval
The Evaluation Pipeline — the complete end-to-end flow