Answer Quality ScorerLLM-as-Judge

TechnicalAccuracyScorer

Evaluates technical accuracy of domain-specific responses. Assesses correctness of technical concepts, procedures, terminology, and quantitative information.

Overview

Evaluates technical accuracy of domain-specific responses. Assesses correctness of technical concepts, procedures, terminology, and quantitative information.

qualityllm-judgetrace-evaluationtechnicaldomain-expertiseaccuracy

Use Cases

  • Accuracy benchmarking and validation

How It Works

This scorer uses LLM-as-Judge technology to evaluate responses. It prompts a large language model with specific evaluation criteria and the content to assess, then analyzes the LLM's judgment to produce a score and detailed reasoning.

Input Schema

ParameterTypeRequiredDescription
output_textstrYesTechnical response to evaluate
input_textstrYesTechnical question
contextdict | strNoDomain knowledge/reference

Output Schema

FieldTypeDescription
scorefloatTechnical accuracy score (0-10)
passedboolTrue if technically accurate
reasoningstrTechnical analysis
metadatadictTechnical issues if any

Score Interpretation

Default threshold: 8/10

9-10ExcellentResponse fully meets all evaluation criteria
7-8GoodResponse meets most criteria with minor issues
5-6FairResponse partially meets criteria, needs improvement
3-4PoorResponse has significant issues
0-2FailingResponse fails to meet basic criteria

Frequently Asked Questions

When should I use this scorer?

Use TechnicalAccuracyScorer when you need to evaluate quality and llm-judge aspects of your AI outputs. It's particularly useful for accuracy benchmarking and validation.

Why doesn't this scorer need expected output?

This scorer evaluates without a reference answer by using the specific input fields listed in its schema. Those fields are evidence for the check; they are not automatically factual ground truth.

Can I customize the threshold?

This scorer's displayed default threshold is 8. Where the eval-job configuration exposes a threshold, review it against the scorer's native scale before saving. The stored pass decision uses that scorer-specific configuration.

Quick Info

CategoryAnswer Quality
Evaluation TypeLLM-as-Judge
Requires Expected OutputNo
Default Threshold8/10

Ready to try TechnicalAccuracyScorer?

Start evaluating your AI agents with Noveum.ai's comprehensive scorer library.

Explore More Scorers

Discover 112 published scorers across 15 categories: 82 Premium, 30 Free, 76 model-backed, and 36 dedicated voice and audio scorers