Accuracy ScorerLLM-as-Judge

AccuracyScorer

Calculates accuracy based on substring or token-level matching between prediction and ground truth. More lenient than ExactMatchScorer, allowing for partial matches.

Overview

Calculates accuracy based on substring or token-level matching between prediction and ground truth. More lenient than ExactMatchScorer, allowing for partial matches.

accuracyrule-basedbenchmarkpartial-matchflexible

Use Cases

  • Accuracy benchmarking and validation

How It Works

This scorer uses LLM-as-Judge technology to evaluate responses. It prompts a large language model with specific evaluation criteria and the content to assess, then analyzes the LLM's judgment to produce a score and detailed reasoning.

Input Schema

ParameterTypeRequiredDescription
predictionstrYesGenerated output
ground_truthstrYesExpected output

Output Schema

FieldTypeDescription
scorefloatPartial match score (0-10)
passedboolTrue if above threshold
reasoningstrMatch analysis
metadatadictToken overlap details

Score Interpretation

Default threshold: 7/10

9-10ExcellentResponse fully meets all evaluation criteria
7-8GoodResponse meets most criteria with minor issues
5-6FairResponse partially meets criteria, needs improvement
3-4PoorResponse has significant issues
0-2FailingResponse fails to meet basic criteria

Frequently Asked Questions

When should I use this scorer?

Use AccuracyScorer when you need to evaluate accuracy and rule-based aspects of your AI outputs. It's particularly useful for accuracy benchmarking and validation.

Why does this scorer need expected output?

This scorer compares the generated output against a known expected result to calculate accuracy metrics.

Can I customize the threshold?

This scorer's displayed default threshold is 7. Where the eval-job configuration exposes a threshold, review it against the scorer's native scale before saving. The stored pass decision uses that scorer-specific configuration.

Quick Info

CategoryAccuracy
Evaluation TypeLLM-as-Judge
Requires Expected OutputYes
Default Threshold7/10

Ready to try AccuracyScorer?

Start evaluating your AI agents with Noveum.ai's comprehensive scorer library.

Explore More Scorers

Discover 112 published scorers across 15 categories: 82 Premium, 30 Free, 76 model-backed, and 36 dedicated voice and audio scorers