Choose Scorers

A scorer measures one property of a dataset item. The platform stores its numeric score, pass verdict, optional error, execution time, and optional reasoning.

Use the live scorer catalog for the current built-in inventory, descriptions, categories, and score interpretation. The catalog is generated from the platform registry and stays current as scorers change.

Start with item evidence

Choose a scorer only after inspecting item JSON. The same application may need separate item boundaries or datasets for response, retrieval, conversation, and complete-agent evaluation.

Field names shown in the catalog describe scorer input concepts. The platform maps item fields such as agent_task, input_text, agent_response, output_text, ground_truth, and retrieved_context into those concepts. Verify a small run because not every scorer registry entry has a complete machine-readable input schema.

Score and threshold behavior

Do not assume every scorer uses the same numeric scale or direction. Use the scorer catalog description and default configuration. The Eval Job threshold determines whether the stored score passes.

Recommendations inspect sampled items and dataset field statistics. They do not prove semantic completeness, validate references, or guarantee that a scorer will run successfully.

Complete-interaction scorers

goal_achievement and conversation_coherence require a complete interaction and a terminal agent_exit signal. agent_exit means the item is complete, not that it succeeded.

The normal ETL item contract does not persist raw trace or trace_data output. Do not select a scorer that requires that raw object unless the active ingestion path explicitly supplies a compatible full-interaction payload. For normal ETL datasets, prefer scorers supported by stored conversation, task, response, tool, retrieval, and terminal fields.

Voice and latency scorers

Audio and timing evidence varies by integration, provider, and mapper. Inspect raw_complete_audio, stt_data, tts_data, latency, and typed metric arrays on the actual item. Then confirm the selected scorer runs on a small sample.

Avoid selecting voice or latency scorers based only on the application framework. Framework tracing does not guarantee that every media or metric field is present.

Create a custom scorer

Open organization scorer settings and create a custom scorer through the guided authoring flow or manual form. Configure its name, description, prompt template, pass threshold, provider, and model.

Persisted templates use simple {identifier} placeholders. Supported placeholders are:

agent_task, agent_response, system_prompt, expected_output,
input_text, output_text, retrieved_context, interaction_context,
extra_context, agent_role, agent_name, ground_truth,
expected_tool_call, tool_calls, parameters_passed, tool_call_results,
retrieval_query, conversation_context, speaker, message, turn_position,
metadata, context, trace

Only reference fields populated by the dataset path. In particular, normal ETL normalization does not persist a raw trace field.

Given the user task:
{agent_task}

Given the agent response:
{agent_response}

Score whether the response resolves the task and explain the result.

The editor warns about placeholders outside the supported catalog. Test the scorer on representative items before adding it to a realtime job.

Practical selection process

1. define the quality question
2. choose the item boundary that contains the needed evidence
3. inspect representative item JSON
4. consult the live scorer catalog
5. add the smallest compatible scorer set
6. run a small filtered evaluation
7. inspect errors and optional reasoning
8. correct the mapper or scorer choice before scaling the run

Continue with Run Eval Jobs or review the dataset item contracts.