Agent
Score whether the agent's tool calls and response adhere to its assigned role and task.
Click for detailsScale: 0–10, pass ≥ 7
Published Scorer Library
112 Published Scorers for Comprehensive AI Evaluation
Evaluate AI agents with a mixed library of model-backed, deterministic, audio-model, and signal-based scorers across 15 categories, with native numeric, boolean, and label outputs.
LLM-as-Judge Technology
Comprehensive Coverage
Reference-Free Options
112
Production scorers
15
Quality categories
76+
LLM-as-Judge metrics
Configure compatible checks from a mixed scorer library, then review each scorer's native result and saved pass decision
Why use Noveum.ai Scorers/Evals?
LLM-as-Judge Technology
The library includes 76 model-backed scorers for checks that benefit from judge-model reasoning; deterministic, audio-model, and signal-based scorers cover other tasks.
Reference-Free Options
Use scorers that do not need expected output when their required task, response, context, or role fields are available. Add reviewed references for checks that require them.
Comprehensive Coverage
112 published scorers covering AI quality from hallucination detection to bias assessment.
Enterprise-Ready
Evaluate representative production data with deployment and support options that can be scoped for enterprise requirements.
Fully Customizable
Create custom scorers for your specific business needs. Extend existing scorers or build from scratch.
Trace-Based Evaluation
Evaluate the workflow operations captured in structured dataset items, including available tool calls, reasoning fields, and multi-turn context.
36 voice & audio scorers
Voice agents, evaluated across captured call stages
A dedicated suite of 36 voice and audio scorers covers speech recognition, synthesis quality, latency, and call dynamics. Input requirements vary by scorer; use mapped production or synthetic call data with the required fields.
Stage 1
Speech-to-Text
Was the caller heard correctly: and fast enough?
SttEfficacyScorerWordAccuracyScorerSttOverSuppressionScorerTranscriptionDelayScorerSttLatencyScorer
Stage 2
Reasoning
Did the model start responding without dead air?
LlmTtftScorerLlmLatencyScorer
Stage 3
Text-to-Speech
Did the agent sound natural, clear, and correct?
MispronunciationScorerAudioBreakageScorerMosScorerToneClarityScorerTtsTtfbScorerTtsLatencyScorer
Stage 4
Call Dynamics
Interruptions, talk ratio, pacing, sentiment, and how the call ended.
E2eLatencyScorerEndOfTurnDelayScorerAiInterruptUserScorerTalkRatioScorerAgentWpmScorerSentimentCsatScorerAppropriateCallTerminationScorer
Reference-free options for checks that do not need expected output
Audio-native judges listen to the actual call
Works with mapped production and synthetic call data
Explore All Scorers
Search and filter 112+ scorers across 15 categories
Showing 112 of 112 scorers
Agent
Evaluates how well the agent maintains and utilizes relevant context throughout the interaction. Assesses whether responses are appropriate given the agent's task, role, and available information. Also useful for RAG context evaluation.
Trace-FriendlyScale: 0–10, pass ≥ 7
Agent
Evaluates the logical flow and coherence of the agent's responses throughout the conversation. Assesses whether the agent maintains consistent reasoning and clear communication across all interactions.
Trace-FriendlyScale: 0–10, pass ≥ 7
Agent
Measures the degree to which the agent successfully accomplishes the stated goals or objectives. Analyzes the complete agent trace to determine if the original goal was achieved and how effectively.
Trace-FriendlyScale: 0–10, pass ≥ 7
Agent
Assesses the accuracy and appropriateness of parameters passed to tools. Verifies that parameter values are sensible and match the task context. Evaluates each tool call's parameters individually.
Trace-FriendlyScale: 0–10, pass ≥ 7
Agent
Measures how effectively the agent progresses toward completing its assigned task. Evaluates logical step sequencing, efficiency of actions taken, and whether the agent is making meaningful progress toward the goal.
Trace-FriendlyScale: 0–10, pass ≥ 7
Agent
Evaluates if tools were used correctly according to their specifications. Checks for proper parameter types, required fields, and valid value ranges. Compares actual tool calls against expected tool call patterns.
Trace-FriendlyScale: 0–10, pass ≥ 7
Agent
Evaluates whether the agent selected appropriate tools for the given task. Assesses if the chosen tools align with the task requirements and whether better alternatives existed. Uses LLM-based analysis to score tool selection quality.
Trace-FriendlyScale: 0–10, pass ≥ 7
Conversational
LLM judge that scores whether the current assistant response follows expected flow from the system prompt and conversation history (0-10 JSON score + reasoning).
Click for detailsScale: 0–10, pass ≥ 7
Conversational
Assesses how relevant each response is to the current conversation context and user query. Evaluates whether responses stay on topic, address the user's actual question, and maintain thematic coherence with the ongoing dialogue.
Trace-FriendlyScale: 0–10, pass ≥ 7
Conversational
A composite scorer that combines multiple conversational evaluation metrics into a single comprehensive assessment. Aggregates knowledge retention, relevancy, completeness, and role adherence scores with configurable weights.
Trace-FriendlyScale: 0–10, pass ≥ 7
Conversational
Measures how well an AI maintains its assigned role or persona throughout a conversation. Evaluates consistency in tone, vocabulary, knowledge boundaries, and behavioral patterns that define the assigned role. Overlaps with safety for persona compliance.
Trace-FriendlyScale: 0–10, pass ≥ 7
Conversational
Evaluates intention fulfillment by analyzing user intentions and fulfillment.
Click for detailsScale: 0–10, pass ≥ 7
Conversational
Evaluates how well an AI maintains and utilizes information across conversation turns. Assesses whether the model remembers and correctly applies information shared earlier in the conversation, avoiding contradictions and maintaining consistency.
Trace-FriendlyScale: 0–10, pass ≥ 7
Conversational
Rule-based repetition scorer using difflib SequenceMatcher similarity between the current assistant message and the prior assistant turn in conversation_context.
Click for detailsScale: 0–10, pass ≥ 7
Conversational
Assesses how well the agent adheres to its defined role and persona throughout the conversation. Evaluates consistency in behavior, tone, and decision-making aligned with the assigned role. Critical for safety and persona compliance.
Trace-FriendlyScale: 0–10, pass ≥ 7
Telephony & Calls
Judges whether the call or session ended appropriately from the transcript.
Click for detailsScale: 0–10, pass ≥ 7
Telephony & Calls
Evaluates logical flow and coherence of assistant turns from the transcript only (no trace required).
Click for detailsScale: 0–10, pass ≥ 7
Telephony & Calls
Detects user disengagement or conversation breakdown (drop-off); returns flow dimensions and drop_off_node_id.
Click for detailsScale: 0–10, pass ≥ 7
Telephony & Calls
Evaluates instruction and role adherence across the full conversation transcript.
Click for detailsScale: 0–10, pass ≥ 7
Telephony & Calls
Evaluates user sentiment and predicted customer satisfaction (CSAT) over the conversation.
Click for detailsScale: 0–10, pass ≥ 7
Voice & Audio
Agent speaking rate (words per minute) from assistant turn text and VAD-gated agent speech time. conversation_context is optional but required for accurate WPM calculation.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Counts agent barge-in events (agent starts speaking while user utterance is open). Zero events → 10.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Mean fundamental frequency (Hz) on the assistant channel using librosa yin, mapped to 1-10 for thresholding.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Silero VAD on stereo conversation audio; scores user-to-agent response latency (1-10, lower latency is better).
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Dual-talk overlap duration from Silero VAD; lower overlap scores higher. Metadata includes overlap_s and overlap_ratio.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
RMS energy of the assistant channel mapped to 1-10 (approx -50 dBFS to -5 dBFS range).
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Scorer for evaluating audio breakage and technical quality in TTS-generated speech.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Detects gibberish or non-coherent assistant speech via multimodal model; metadata may include is_gibberish.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Scorer for evaluating mispronunciation in TTS-generated speech.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Mean Opinion Score style rubric (1-10) on assistant speech using a multimodal audio-capable model.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Pronunciation quality judged from assistant audio only (no reference transcript).
Click for detailsScale: 1–10, higher is better
Voice & Audio
Scorer for evaluating overlapping speech in voice conversations.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Scores transcript fidelity vs spoken audio (1-10). Parses JSON score and reasoning; metadata may include wer_estimate, omissions, hallucinations, substitutions, severity when returned by the model.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Pairwise over-suppression detector. Two physics-grounded signals: musical noise (per-bin temporal-kurtosis ratio of cleaned vs raw in the speech band; Esch & Vary, ICASSP 2009) and flattened modulation (loss of 4-16 Hz syllable-rate envelope energy). No LLM, no GPU: numpy + librosa + soundfile, ~30 ms per pair.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Agent-to-user speech time ratio from VAD intervals. Metadata includes talk_ratio, user_speech_s, and agent_speech_s.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Evaluates tone and clarity of assistant speech from audio alone; returns 1-10 with reasoning.
Click for detailsScale: 1–10, higher is better
Voice & Audio
Counts user barge-in events (user starts speaking while agent utterance is open). Metadata includes stop_time_after_interruption_s.
Click for detailsScale: 0–10, measured from audio
Voice & Audio
Scorer for evaluating word accuracy in TTS-generated speech.
Click for detailsScale: 1–10, higher is better
Voice Latency
Scorer for End-to-End (E2E) latency.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for End-of-Turn detection delay.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for LLM latency in milliseconds.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for LLM Time to First Token (TTFT).
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for User Turn Completion callback delay.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for STT input audio duration.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for STT latency.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for STT processing duration.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for STT transcription delay.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for TTS generated audio duration.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for TTS synthesis duration.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for TTS latency.
Click for detailsScale: ms → 0–10 (10 = ideal)
Voice Latency
Scorer for TTS Time to First Byte (TTFB).
Click for detailsScale: ms → 0–10 (10 = ideal)
RAG
Combines multiple retrieval scorers with weighted averaging for comprehensive RAG evaluation. Allows configurable weights for different metrics to compute a single aggregate score.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Evaluates how relevant the generated answer is to the input question. Assesses whether the response directly addresses what was asked, stays on topic, and provides useful information. Fundamental RAG metric.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Comprehensive bias detection scorer that evaluates content for multiple bias types simultaneously. Detects gender, racial, cultural, political, and other forms of bias in AI-generated responses with detailed severity information.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Computes precision for retrieved chunks using numerical relevance scoring. Precision = (Number of relevant chunks retrieved) ÷ (Total number of chunks retrieved). Uses LLM-based relevance assessment.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Measures precision of retrieved context by evaluating what fraction of retrieved chunks are actually relevant to answering the question. High precision indicates mostly useful information without excessive irrelevant content.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Computes recall for retrieved chunks by estimating total relevant chunks available. Recall = (Number of relevant chunks retrieved) ÷ (Estimated total relevant chunks). Uses LLM-based estimation.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Measures recall of retrieved context by estimating what fraction of relevant information was successfully retrieved. Helps identify gaps in retrieval coverage.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Measures how faithful the generated answer is to the provided context. Critical RAG metric that evaluates whether the answer is grounded in and supported by retrieved documents, detecting hallucinations and unsupported claims.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Implements the RAGAS (Retrieval Augmented Generation Assessment) framework for comprehensive RAG evaluation. Combines answer relevancy, faithfulness, context precision, and context recall metrics.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Evaluates diversity of retrieved context chunks using cosine distance between embeddings. Higher scores indicate more diverse retrieved content, reducing redundancy in results.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
F1 score combining precision and recall for contextual evaluation. Computes harmonic mean of precision and recall scores for balanced retrieval assessment.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Computes ranking metrics for retrieved context including MRR (Mean Reciprocal Rank), NDCG, and MAP. Assesses how well the retrieval system ranks relevant documents higher than irrelevant ones.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Computes semantic similarity between query and retrieved context using sentence embeddings. Uses cosine similarity between embeddings with fallback to text-based similarity if embedding model unavailable.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG
Evaluates quality of source attribution and citations in answers.
Click for detailsScale: 0–10, pass ≥ 7
Multi-Context
Evaluates how well an answer handles contradictory information across multiple context chunks. Performs pairwise comparison to detect conflicts and assesses resolution effectiveness.
Trace-FriendlyScale: 0–10, pass ≥ 7
Multi-Context
Evaluates whether the provided context contains all necessary information to fully support the answer. Assesses if context is complete enough to answer the query comprehensively, helping evaluate retrieval system effectiveness.
Trace-FriendlyScale: 0–10, pass ≥ 7
Multi-Context
Evaluates consistency of an answer across multiple context chunks. Identifies cases where the answer may be faithful to some contexts but inconsistent with others.
Trace-FriendlyScale: 0–10, pass ≥ 7
Multi-Context
Enhanced faithfulness detection with fine-grained claim-by-claim analysis. Performs two-stage evaluation: first extracting individual factual claims from the answer, then verifying each claim against context. Particularly effective for detecting partial hallucinations.
Trace-FriendlyScale: 0–10, pass ≥ 7
Multi-Context
Enhanced faithfulness detection with fine-grained claim-by-claim analysis.
Click for detailsScale: 0–10, pass ≥ 7
Multi-Context
Evaluates how well an answer is grounded in and supported by the provided context. Ensures claims can be traced back to source material. Essential for RAG reliability.
Trace-FriendlyScale: 0–10, pass ≥ 7
Multi-Context
Evaluates how well an answer appropriately weights and prioritizes multiple context chunks. Assesses whether the answer correctly emphasizes the most relevant context.
Trace-FriendlyScale: 0–10, pass ≥ 7
Hallucination
Extracts individual factual claims from output text, then verifies each claim against context. Provides detailed verification results including supporting/contradicting evidence and confidence scores.
Trace-FriendlyScale: 0–10, pass ≥ 7
Hallucination
Evaluates factual accuracy of output text against provided context. Assesses whether statements in the answer align with facts in the context, identifying inaccuracies and unsupported claims.
Trace-FriendlyScale: 0–10, pass ≥ 7
Hallucination
Detects hallucinations and factual inconsistencies in generated answers by comparing against source documents and knowledge base, flagging unsupported or contradictory claims.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates completeness and coverage of AI-generated answers. Assesses whether the answer addresses all aspects of the question and uses available context effectively.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates quality of source attribution and citations in answers. Assesses accuracy, completeness, specificity, and appropriateness of source references.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates clarity and coherence of AI-generated responses. Assesses language clarity, structure, logical flow, and consistency. Measures how well the response communicates its message.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates information density and richness of AI-generated responses. Assesses how information-rich an answer is relative to the question, measuring content value and filler content.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates alignment between questions and answers. Assesses how well the answer directly addresses the question, stays relevant, and provides appropriate detail level.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates overall quality of RAG-generated answers. Assesses question-answer alignment, measuring how well the answer directly addresses the question and provides appropriate detail.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates technical accuracy of domain-specific responses. Assesses correctness of technical concepts, procedures, terminology, and quantitative information.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates consistency and accuracy of domain-specific terminology. Assesses correct use of domain terms, consistent terminology throughout, and adherence to domain standards.
Trace-FriendlyScale: 0–10, pass ≥ 7
Answer Quality
Evaluates tone consistency and appropriateness of responses. Assesses whether tone matches domain, remains consistent throughout, and aligns with conversation history.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG Pipeline
Analyzes error propagation across pipeline stages.
Click for detailsScale: 0–10, pass ≥ 7
RAG Pipeline
Analyzes latency across pipeline stages.
Click for detailsScale: 0–10, pass ≥ 7
RAG Pipeline
Evaluates how well pipeline stages work together.
Click for detailsScale: 0–10, pass ≥ 7
RAG Pipeline
Evaluates query understanding and processing quality. Assesses clarity, intent detection, preprocessing effectiveness, specificity, complexity, and ambiguity of input queries with configurable weights.
Trace-FriendlyScale: 0–10, pass ≥ 7
RAG Pipeline
Evaluates reranking stage effectiveness.
Click for detailsScale: 0–10, pass ≥ 7
RAG Pipeline
Analyzes resource utilization across pipeline stages.
Click for detailsScale: 0–10, pass ≥ 7
RAG Pipeline
Enhanced retrieval stage performance evaluation combining precision, recall, ranking, similarity, and diversity metrics. Provides comprehensive assessment of the retrieval component.
Trace-FriendlyScale: 0–10, pass ≥ 7
Safety
Evaluates whether the model appropriately refuses to answer harmful or inappropriate requests. Assesses if the model correctly identifies dangerous queries and provides appropriate refusals. Important for compliance and guardrails.
Trace-FriendlyScale: 0–10, pass ≥ 7
Safety
Evaluates content against moderation guidelines and policies. Checks for violations across multiple categories including violence, hate speech, adult content, and self-harm. Essential for platform safety.
Trace-FriendlyScale: 0–10, pass ≥ 7
Safety
Detects specific safety policy violations in generated content. More granular than general moderation, focusing on specific prohibited content types with detailed violation reports and severity levels.
Trace-FriendlyScale: 0–10, pass ≥ 7
Safety
Evaluates whether generated content contains harmful advice or dangerous recommendations. Detects advice that could cause physical, financial, legal, or psychological harm to users.
Trace-FriendlyScale: 0–10, pass ≥ 7
Safety
Measures the level of toxic content in generated text. Evaluates text for various forms of toxicity including insults, threats, profanity, and harassment with specific toxic element identification.
Trace-FriendlyScale: 0–10, pass ≥ 7
Bias & Fairness
Assesses cultural sensitivity and appropriateness of generated content. Evaluates whether text respects cultural differences, avoids cultural appropriation, and maintains sensitivity toward diverse cultural practices and beliefs.
Trace-FriendlyScale: 0–10, pass ≥ 7
Bias & Fairness
Evaluates content for age-based discrimination or stereotypes. Identifies ageist language, assumptions about capabilities based on age, or stereotypical portrayals of different age groups.
Trace-FriendlyScale: 0–10, pass ≥ 7
Bias & Fairness
Evaluates whether the generated content contains gender-based biases or stereotypes. Analyzes text for implicit or explicit gender discrimination, stereotypical portrayals, or language that unfairly associates traits or roles with specific genders.
Trace-FriendlyScale: 0–10, pass ≥ 7
Bias & Fairness
Detects racial or ethnic biases in generated content. Uses an LLM to identify stereotypes, discriminatory language, or unfair associations based on race or ethnicity. Examines both explicit statements and subtle linguistic patterns.
Trace-FriendlyScale: 0–10, pass ≥ 7
Accuracy
Calculates accuracy based on substring or token-level matching between prediction and ground truth. More lenient than ExactMatchScorer, allowing for partial matches.
Click for detailsScale: 0–10, pass ≥ 7
Accuracy
Evaluates whether the prediction exactly matches the ground truth. Strictest form of accuracy measurement for tasks requiring exact output like code generation or classification labels.
Click for detailsScale: 0–10, pass ≥ 7
Accuracy
Computes token-level F1 score between prediction and ground truth. Balances precision and recall, useful for extractive tasks and named entity recognition.
Click for detailsScale: 0–10, pass ≥ 7
Accuracy
Evaluates prediction accuracy against multiple acceptable patterns or answers. Ideal for tasks where multiple correct answers exist, with regex support for flexible pattern matching.
Click for detailsScale: 0–10, pass ≥ 7
NLP Metrics
Computes BLEU (Bilingual Evaluation Understudy) score between prediction and ground truth. Measures n-gram precision with a brevity penalty, useful for machine translation and text summarization evaluation.
Click for detailsScale: 0–10, pass ≥ 7
NLP Metrics
Computes similarity based on Levenshtein (edit) distance between prediction and ground truth. Measures the minimum number of single-character edits needed to transform one string into another.
Click for detailsScale: 0–10, pass ≥ 7
NLP Metrics
Computes ROUGE scores for text summarization evaluation. Calculates ROUGE-1 (unigram), ROUGE-2 (bigram), and ROUGE-L (longest common subsequence) metrics for comprehensive content overlap analysis.
Click for detailsScale: 0–10, pass ≥ 7
Format
Validates whether the prediction contains a valid email address format. Uses regular expression matching to check for properly formatted email addresses in standalone strings or text.
Trace-FriendlyScale: 0–10, pass ≥ 7
Format
Validates whether the prediction is valid JSON. This rule-based scorer attempts to parse the prediction as JSON and returns a perfect score if parsing succeeds. Useful for structured output tasks.
Trace-FriendlyScale: 0–10, pass ≥ 7
Format
Validates that JSON output conforms to a specified JSON Schema. Checks JSON validity and ensures the structure matches expected fields, types, and constraints defined in a provided schema.
Trace-FriendlyScale: 0–10, pass ≥ 7
Format
Validates that URLs in the prediction are properly formatted and optionally accessible. Extracts URLs from text, validates their format, and can optionally check if they resolve to valid resources.
Trace-FriendlyScale: 0–10, pass ≥ 7
Find the right scorers for your specific use case
Do You Have the Scorers You Need?
RAG System Evaluation
- AnswerRelevancyScorer
- FaithfulnessScorer
- ContextualPrecisionScorer
- ContextualRecallScorer
- RAGASScorer
Safety & Compliance
- ToxicityScorer
- ContentSafetyViolationScorer
- IsHarmfulAdviceScorer
- ContentModerationScorer
- AnswerRefusalScorer
Bias Detection
- NoGenderBiasScorer
- NoRacialBiasScorer
- NoAgeBiasScorer
- CulturalSensitivityScorer
- BiasDetectionScorer
Everything you need to know about Noveum.ai scorers
Frequently Asked Questions
What are evaluation scorers?
Evaluation scorers assess different aspects of AI agent behavior. Depending on the scorer, they use model-backed judging, deterministic rules, audio models, or measured signals, and some require particular dataset fields or expected outputs.
How many scorers does Noveum.ai have?
Noveum.ai publishes 112 scorers across 15 categories: 82 Premium, 30 Free, 76 model-backed, and 36 dedicated voice and audio scorers. Categories include RAG, hallucination, bias, safety, voice, and more.
Do I need to provide expected outputs?
Not always. Requirements vary by scorer. Many checks are reference-free; instruction and role checks can use a system prompt, while factual checks may require retrieved context, a reference answer, or tool evidence.
Can I create custom scorers?
Yes. Create a custom model-judged criterion from a rubric prompt, map its placeholders to dataset fields, choose a judge model, and set the pass threshold.
Which scorers should I use for my use case?
It depends on your specific needs. For RAG systems, use our RAG scorers (Faithfulness, ContextualPrecision, etc.). For agent evaluation, use Agent scorers. For safety compliance, use Safety and Bias scorers. Browse our categories to find the right fit.
How do scorers impact performance?
Evaluation time and credit cost vary by scorer, model, item count, and data shape. Noveum runs configured eval jobs asynchronously, supports bounded batches and filters, and shows a credit estimate before a manual run.
Can I use scorers in my own code?
Use Noveum eval-job APIs to start configured evaluations from CI or another service, then poll the run and read its results. The tracing SDK records application behavior; it does not execute the entire scorer catalog inline.
What's the difference between LLM-based and rule-based scorers?
LLM-based scorers use advanced language models to evaluate nuanced quality aspects like relevancy and faithfulness. Rule-based scorers use deterministic algorithms for objective metrics like JSON validation or exact matching. Both have their place depending on your needs.
Start Free
Ready to evaluate your AI agents?
Start with 112 published scorers. No credit card required.