Dataset Item Contracts
Understand ETL mapper output, REST item envelopes, normalized content, and provenance.
Noveum uses related but distinct contracts for mapper output, REST input, and stored dataset items. Keeping those layers separate prevents fields from being placed in the wrong object.
Contract overview
| Layer | Shape | Purpose |
|---|---|---|
| ETL mapper output | A flat object, a list of flat objects, or no items | Converts one complete source trace into evaluation units |
| REST item envelope | item_type, content, optional metadata, trace_id, and span_id | Adds already-normalized items through the API |
| Stored item | Envelope, normalized content, provenance, version, and scores | Supports dataset reads, filtering, evaluation, and versioning |
There is no automatic synchronization between agent_task and input_text, or between agent_response and output_text. Populate the fields required by the selected scorer.
ETL mapper output
An ETL mapper receives one complete trace, including trace attributes, metadata, and span attributes. It can return:
- no object for an irrelevant or incomplete trace
- one object for a complete request or conversation
- a list for turn-level or operation-level evaluation
Use a flat output shape:
[
{
"item_type": "agent",
"trace_id": "trace-abc123",
"span_id": "span-final",
"agent_name": "support-agent",
"agent_role": "Customer support specialist",
"agent_task": "How do I cancel my subscription?",
"agent_response": "Open Settings, choose Billing, then select Cancel plan.",
"system_prompt": "Answer support questions from the supplied policy.",
"retrieval_query": ["cancel subscription"],
"retrieved_context": [["Open Settings > Billing to cancel a plan."]],
"exit_status": "completed",
"agent_exit": true,
"metadata": {
"service_version": "support-agent-v3"
}
}
]Mapper normalization moves content fields into the item content, keeps metadata on the envelope, and derives provenance from trace_id and span_id. Objects without a task, response, output, or message are dropped.
The mapper must explicitly read the application's actual trace keys. Noveum does not infer provider-specific paths from arbitrary attributes.
REST item envelope
POST /api/v1/datasets/{datasetSlug}/items accepts one or more normalized items:
{
"items": [
{
"item_type": "agent",
"trace_id": "trace-abc123",
"span_id": "span-final",
"content": {
"agent_task": "How do I cancel my subscription?",
"agent_response": "Open Settings, choose Billing, then select Cancel plan.",
"system_prompt": "Answer support questions from the supplied policy.",
"ground_truth": "Settings > Billing > Cancel plan",
"agent_exit": "true"
},
"metadata": {
"service_version": "support-agent-v3"
}
}
]
}REST callers place provenance on the envelope, not inside content. agent_exit is currently accepted as a string through this endpoint.
Verified content fields
Core interaction
| Field | Purpose |
|---|---|
agent_name, agent_role | Agent identity and intended role |
agent_task, input_text | Task or user input |
agent_response, output_text, message | Generated or conversational output |
system_prompt | Instructions active for the interaction |
ground_truth, expected_output | Reference output when available |
user_id, session_id, turn_id, conversation_id | Correlation within the dataset item |
Tools and retrieval
| Field | Purpose |
|---|---|
tools_available | Offered tool names, descriptions, and schemas |
tool_calls | Calls selected by the model |
parameters_passed | Arguments supplied to tools |
tool_call_results | Results and errors returned by tools |
expected_tool_call | Expected tool identifier or serialized expectation |
retrieval_query | Ordered retrieval queries |
retrieved_context | Ordered context returned for those queries |
Conversation and terminal state
| Field | Purpose |
|---|---|
conversation_context | Structured prior turns and conversation context |
speaker, message | Speaker and content for a turn item |
exit_status | Terminal result such as completed, error, timeout, or cancelled |
agent_exit | Whether the item represents a complete terminal execution |
agent_exit means completeness, not success. A failed terminal execution can still have agent_exit set. Use exit_status to distinguish the outcome. Set agent_exit only on an aggregate item that contains the complete interaction. Python mapper output may use True; REST input currently uses "true".
Evaluation and custom context
| Field | Purpose |
|---|---|
evaluation_context | Structured evidence supplied for evaluation |
criteria | Item-specific criteria when supported by the scorer path |
quality_score | Existing quality value supplied with the item |
validation_status, validation_errors | Upstream validation state |
tags, custom_attributes | Searchable or application-specific context |
Voice and timing
The item contract can store stt_data, tts_data, raw_complete_audio, latency, vad_metrics, stt_metrics, tts_metrics, llm_metrics, and eou_metrics.
These fields are flexible JSON payloads. Their presence and inner shape depend on the integration and mapper. Validate the requirements of a selected audio or latency scorer against representative items instead of assuming every voice integration emits the same metrics.
Provenance and readback
Stored items expose source_trace_id and source_span_id. They come from the ETL or REST envelope and link an item back to its source trace and span.
The normal ETL item path does not persist raw trace or trace_data fields. Scorers that require a complete raw-trace object are not compatible with a normal ETL dataset unless another supported ingestion path supplies the exact full-interaction contract.
Design items for scorers
Before running an Eval Job:
- choose the behavior and item boundary to measure
- inspect actual item JSON
- verify every selected scorer input is populated
- test a small run and inspect errors or missing results
- adjust the mapper instead of relying on field aliases
Continue with the scorer compatibility guide or map source traces with ETL.