AgentRoleAdherenceScorer Score whether the agent's tool calls and response adhere to its assigned role and task.
agent
Click for details Threshold: 7
ContextRelevancyScorer Evaluates how well the agent maintains and utilizes relevant context throughout the interaction. Assesses whether responses are appropriate given the agent's task, role, and available information. Also useful for RAG context evaluation.
agent rag context +3
Threshold: 7 ConversationCoherenceScorer Evaluates the logical flow and coherence of the agent's responses throughout the conversation. Assesses whether the agent maintains consistent reasoning and clear communication across all interactions.
agent conversational llm-judge +3
Threshold: 7 GoalAchievementScorer Measures the degree to which the agent successfully accomplishes the stated goals or objectives. Analyzes the complete agent trace to determine if the original goal was achieved and how effectively.
agent workflow llm-judge +3
Threshold: 7 ParameterCorrectnessScorer Assesses the accuracy and appropriateness of parameters passed to tools. Verifies that parameter values are sensible and match the task context. Evaluates each tool call's parameters individually.
agent tool-usage llm-judge +2
Threshold: 7 TaskProgressionScorer Measures how effectively the agent progresses toward completing its assigned task. Evaluates logical step sequencing, efficiency of actions taken, and whether the agent is making meaningful progress toward the goal.
agent workflow llm-judge +2
Threshold: 7 ToolCorrectnessScorer Evaluates if tools were used correctly according to their specifications. Checks for proper parameter types, required fields, and valid value ranges. Compares actual tool calls against expected tool call patterns.
agent tool-usage llm-judge +2
Threshold: 7 ToolRelevancyScorer Evaluates whether the agent selected appropriate tools for the given task. Assesses if the chosen tools align with the task requirements and whether better alternatives existed. Uses LLM-based analysis to score tool selection quality.
agent tool-usage llm-judge +1
Threshold: 7 ConversationFlowAdherenceScorer LLM judge that scores whether the current assistant response follows expected flow from the system prompt and conversation history (0-10 JSON score + reasoning).
repetition_and_flow conversational
Click for details Threshold: 7
ConversationRelevancyScorer Assesses how relevant each response is to the current conversation context and user query. Evaluates whether responses stay on topic, address the user's actual question, and maintain thematic coherence with the ongoing dialogue.
conversational rag llm-judge +3
Threshold: 7 ConversationalMetricsScorer A composite scorer that combines multiple conversational evaluation metrics into a single comprehensive assessment. Aggregates knowledge retention, relevancy, completeness, and role adherence scores with configurable weights.
conversational quality llm-judge +3
Threshold: 7 ConversationalRoleAdherenceScorer Measures how well an AI maintains its assigned role or persona throughout a conversation. Evaluates consistency in tone, vocabulary, knowledge boundaries, and behavioral patterns that define the assigned role. Overlaps with safety for persona compliance.
conversational agent safety +4
Threshold: 7 IntentionFulfillmentScorer Evaluates intention fulfillment by analyzing user intentions and fulfillment.
conversational
Click for details Threshold: 7
KnowledgeRetentionScorer Evaluates how well an AI maintains and utilizes information across conversation turns. Assesses whether the model remembers and correctly applies information shared earlier in the conversation, avoiding contradictions and maintaining consistency.
conversational memory llm-judge +3
Threshold: 7 RepetitionFuzzyMatchScorer Rule-based repetition scorer using difflib SequenceMatcher similarity between the current assistant message and the prior assistant turn in conversation_context.
repetition_and_flow conversational
Click for details Threshold: 7
RoleAdherenceScorer Assesses how well the agent adheres to its defined role and persona throughout the conversation. Evaluates consistency in behavior, tone, and decision-making aligned with the assigned role. Critical for safety and persona compliance.
agent conversational safety +4
Threshold: 7 AppropriateCallTerminationScorer Judges whether the call or session ended appropriately from the transcript.
telephony
Click for details Threshold: 7
ConversationContextCoherenceScorer Evaluates logical flow and coherence of assistant turns from the transcript only (no trace required).
telephony
Click for details Threshold: 7
DropOffNodeScorer Detects user disengagement or conversation breakdown (drop-off); returns flow dimensions and drop_off_node_id.
telephony
Click for details Threshold: 7
InstructionAdherenceScorer Evaluates instruction and role adherence across the full conversation transcript.
telephony
Click for details Threshold: 7
SentimentCsatScorer Evaluates user sentiment and predicted customer satisfaction (CSAT) over the conversation.
telephony
Click for details Threshold: 7
AgentWpmScorer Agent speaking rate (words per minute) from assistant turn text and VAD-gated agent speech time. conversation_context is optional but required for accurate WPM calculation.
livekit_conversation audio
Click for details Threshold: 7
AiInterruptUserScorer Counts agent barge-in events (agent starts speaking while user utterance is open). Zero events → 10.
livekit_conversation audio
Click for details Threshold: 7
AssistantAveragePitchScorer Mean fundamental frequency (Hz) on the assistant channel using librosa yin, mapped to 1-10 for thresholding.
livekit_conversation audio
Click for details Threshold: 7
AssistantLatencyScorer Silero VAD on stereo conversation audio; scores user-to-agent response latency (1-10, lower latency is better).
livekit_conversation audio
Click for details Threshold: 7
AssistantOverlapScorer Dual-talk overlap duration from Silero VAD; lower overlap scores higher. Metadata includes overlap_s and overlap_ratio.
livekit_conversation audio
Click for details Threshold: 7
AssistantVolumeScorer RMS energy of the assistant channel mapped to 1-10 (approx -50 dBFS to -5 dBFS range).
livekit_conversation audio
Click for details Threshold: 7
AudioBreakageScorer Scorer for evaluating audio breakage and technical quality in TTS-generated speech.
audio
Click for details Threshold: 7
GibberishScorer Detects gibberish or non-coherent assistant speech via multimodal model; metadata may include is_gibberish.
livekit_voice_quality audio
Click for details Threshold: 7
MispronunciationScorer Scorer for evaluating mispronunciation in TTS-generated speech.
audio
Click for details Threshold: 7
MosScorer Mean Opinion Score style rubric (1-10) on assistant speech using a multimodal audio-capable model.
livekit_voice_quality audio
Click for details Threshold: 7
PronunciationAudioScorer Pronunciation quality judged from assistant audio only (no reference transcript).
livekit_voice_quality audio
Click for details Threshold: 7
SpeakingOverUserScorer Scorer for evaluating overlapping speech in voice conversations.
audio
Click for details Threshold: 7
SttEfficacyScorer Scores transcript fidelity vs spoken audio (1-10). Parses JSON score and reasoning; metadata may include wer_estimate, omissions, hallucinations, substitutions, severity when returned by the model.
stt_efficacy audio
Click for details Threshold: 7
SttOverSuppressionScorer Pairwise over-suppression detector. Two physics-grounded signals: musical noise (per-bin temporal-kurtosis ratio of cleaned vs raw in the speech band; Esch & Vary, ICASSP 2009) and flattened modulation (loss of 4-16 Hz syllable-rate envelope energy). No LLM, no GPU — numpy + librosa + soundfile, ~30 ms per pair.
audio
Click for details Threshold: 7
TalkRatioScorer Agent-to-user speech time ratio from VAD intervals. Metadata includes talk_ratio, user_speech_s, and agent_speech_s.
livekit_conversation audio
Click for details Threshold: 7
ToneClarityScorer Evaluates tone and clarity of assistant speech from audio alone; returns 1-10 with reasoning.
livekit_voice_quality audio
Click for details Threshold: 7
UserInterruptAgentScorer Counts user barge-in events (user starts speaking while agent utterance is open). Metadata includes stop_time_after_interruption_s.
livekit_conversation audio
Click for details Threshold: 7
WordAccuracyScorer Scorer for evaluating word accuracy in TTS-generated speech.
audio
Click for details Threshold: 7
E2eLatencyScorer Scorer for End-to-End (E2E) latency.
latency
Click for details Threshold: 7
EndOfTurnDelayScorer Scorer for End-of-Turn detection delay.
latency
Click for details Threshold: 7
LlmLatencyScorer Scorer for LLM latency in milliseconds.
latency
Click for details Threshold: 7
LlmTtftScorer Scorer for LLM Time to First Token (TTFT).
latency
Click for details Threshold: 7
OnUserTurnCompletedDelayScorer Scorer for User Turn Completion callback delay.
latency
Click for details Threshold: 7
SttAudioDurationScorer Scorer for STT input audio duration.
latency
Click for details Threshold: 7
SttLatencyScorer Scorer for STT latency.
latency
Click for details Threshold: 7
SttProcessingDurationScorer Scorer for STT processing duration.
latency
Click for details Threshold: 7
TranscriptionDelayScorer Scorer for STT transcription delay.
latency
Click for details Threshold: 7
TtsAudioDurationScorer Scorer for TTS generated audio duration.
latency
Click for details Threshold: 7
TtsDurationScorer Scorer for TTS synthesis duration.
latency
Click for details Threshold: 7
TtsLatencyScorer Scorer for TTS latency.
latency
Click for details Threshold: 7
TtsTtfbScorer Scorer for TTS Time to First Byte (TTFB).
latency
Click for details Threshold: 7
AggregateRAGScorer Combines multiple retrieval scorers with weighted averaging for comprehensive RAG evaluation. Allows configurable weights for different metrics to compute a single aggregate score.
rag rule-based trace-evaluation +3
Threshold: 7 AnswerRelevancyScorer Evaluates how relevant the generated answer is to the input question. Assesses whether the response directly addresses what was asked, stays on topic, and provides useful information. Fundamental RAG metric.
rag quality llm-judge +3
Threshold: 7 BiasDetectionScorer Comprehensive bias detection scorer that evaluates content for multiple bias types simultaneously. Detects gender, racial, cultural, political, and other forms of bias in AI-generated responses with detailed severity information.
bias safety llm-judge +4
Threshold: 7 ContextualPrecisionScorerPP Computes precision for retrieved chunks using numerical relevance scoring. Precision = (Number of relevant chunks retrieved) ÷ (Total number of chunks retrieved). Uses LLM-based relevance assessment.
rag llm-judge trace-evaluation +3
Threshold: 7 ContextualPrecisionScorer Measures precision of retrieved context by evaluating what fraction of retrieved chunks are actually relevant to answering the question. High precision indicates mostly useful information without excessive irrelevant content.
rag llm-judge trace-evaluation +3
Threshold: 7 ContextualRecallScorerPP Computes recall for retrieved chunks by estimating total relevant chunks available. Recall = (Number of relevant chunks retrieved) ÷ (Estimated total relevant chunks). Uses LLM-based estimation.
rag llm-judge trace-evaluation +3
Threshold: 7 ContextualRecallScorer Measures recall of retrieved context by estimating what fraction of relevant information was successfully retrieved. Helps identify gaps in retrieval coverage.
rag llm-judge trace-evaluation +3
Threshold: 7 FaithfulnessScorer Measures how faithful the generated answer is to the provided context. Critical RAG metric that evaluates whether the answer is grounded in and supported by retrieved documents, detecting hallucinations and unsupported claims.
rag hallucination safety +4
Threshold: 7 RAGASScorer Implements the RAGAS (Retrieval Augmented Generation Assessment) framework for comprehensive RAG evaluation. Combines answer relevancy, faithfulness, context precision, and context recall metrics.
rag llm-judge trace-evaluation +3
Threshold: 7 RetrievalDiversityScorer Evaluates diversity of retrieved context chunks using cosine distance between embeddings. Higher scores indicate more diverse retrieved content, reducing redundancy in results.
rag rule-based trace-evaluation +3
Threshold: 6 RetrievalF1Scorer F1 score combining precision and recall for contextual evaluation. Computes harmonic mean of precision and recall scores for balanced retrieval assessment.
rag accuracy rule-based +4
Threshold: 7 RetrievalRankingScorer Computes ranking metrics for retrieved context including MRR (Mean Reciprocal Rank), NDCG, and MAP. Assesses how well the retrieval system ranks relevant documents higher than irrelevant ones.
rag rule-based trace-evaluation +4
Threshold: 7 SemanticSimilarityScorer Computes semantic similarity between query and retrieved context using sentence embeddings. Uses cosine similarity between embeddings with fallback to text-based similarity if embedding model unavailable.
rag accuracy rule-based +4
Threshold: 7 SourceAttributionScorer Evaluates quality of source attribution and citations in answers.
rag
Click for details Threshold: 7
ConflictResolutionScorer Evaluates how well an answer handles contradictory information across multiple context chunks. Performs pairwise comparison to detect conflicts and assesses resolution effectiveness.
multi-context rag llm-judge +3
Threshold: 7 ContextCompletenessScorer Evaluates whether the provided context contains all necessary information to fully support the answer. Assesses if context is complete enough to answer the query comprehensively, helping evaluate retrieval system effectiveness.
multi-context completeness retrieval +2
Threshold: 7 ContextConsistencyScorer Evaluates consistency of an answer across multiple context chunks. Identifies cases where the answer may be faithful to some contexts but inconsistent with others.
multi-context rag llm-judge +3
Threshold: 7 ContextFaithfulnessScorerPP Enhanced faithfulness detection with fine-grained claim-by-claim analysis. Performs two-stage evaluation: first extracting individual factual claims from the answer, then verifying each claim against context. Particularly effective for detecting partial hallucinations.
multi-context rag hallucination +4
Threshold: 7 ContextFaithfulnessScorer Enhanced faithfulness detection with fine-grained claim-by-claim analysis.
multi-context rag
Click for details Threshold: 7
ContextGroundednessScorer Evaluates how well an answer is grounded in and supported by the provided context. Ensures claims can be traced back to source material. Essential for RAG reliability.
multi-context rag hallucination +4
Threshold: 7 ContextPrioritizationScorer Evaluates how well an answer appropriately weights and prioritizes multiple context chunks. Assesses whether the answer correctly emphasizes the most relevant context.
multi-context rag llm-judge +3
Threshold: 7 ClaimVerificationScorer Extracts individual factual claims from output text, then verifies each claim against context. Provides detailed verification results including supporting/contradicting evidence and confidence scores.
hallucination rag llm-judge +4
Threshold: 7 FactualAccuracyScorer Evaluates factual accuracy of output text against provided context. Assesses whether statements in the answer align with facts in the context, identifying inaccuracies and unsupported claims.
hallucination rag safety +4
Threshold: 7 HallucinationDetectionScorer Detects hallucinations and factual inconsistencies in generated answers by comparing against source documents and knowledge base, flagging unsupported or contradictory claims.
hallucination rag safety +4
Threshold: 7 AnswerCompletenessScorer Evaluates completeness and coverage of AI-generated answers. Assesses whether the answer addresses all aspects of the question and uses available context effectively.
quality rag conversational +4
Threshold: 7 CitationQualityScorer Evaluates quality of source attribution and citations in answers. Assesses accuracy, completeness, specificity, and appropriateness of source references.
quality rag llm-judge +4
Threshold: 6 ClarityAndCoherenceScorer Evaluates clarity and coherence of AI-generated responses. Assesses language clarity, structure, logical flow, and consistency. Measures how well the response communicates its message.
quality conversational llm-judge +3
Threshold: 7 InformationDensityScorer Evaluates information density and richness of AI-generated responses. Assesses how information-rich an answer is relative to the question, measuring content value and filler content.
quality rag llm-judge +3
Threshold: 6 QuestionAnswerAlignmentScorer Evaluates alignment between questions and answers. Assesses how well the answer directly addresses the question, stays relevant, and provides appropriate detail level.
quality rag llm-judge +3
Threshold: 7 RAGAnswerQualityScorer Evaluates overall quality of RAG-generated answers. Assesses question-answer alignment, measuring how well the answer directly addresses the question and provides appropriate detail.
quality rag llm-judge +3
Threshold: 7 TechnicalAccuracyScorer Evaluates technical accuracy of domain-specific responses. Assesses correctness of technical concepts, procedures, terminology, and quantitative information.
quality llm-judge trace-evaluation +3
Threshold: 8 TerminologyConsistencyScorer Evaluates consistency and accuracy of domain-specific terminology. Assesses correct use of domain terms, consistent terminology throughout, and adherence to domain standards.
quality llm-judge trace-evaluation +3
Threshold: 7 ToneConsistencyScorer Evaluates tone consistency and appropriateness of responses. Assesses whether tone matches domain, remains consistent throughout, and aligns with conversation history.
quality conversational llm-judge +3
Threshold: 7 ErrorPropagationScorer Analyzes error propagation across pipeline stages.
pipeline
Click for details Threshold: 7
LatencyAnalysisScorer Analyzes latency across pipeline stages.
pipeline
Click for details Threshold: 7
PipelineCoordinationScorer Evaluates how well pipeline stages work together.
pipeline
Click for details Threshold: 7
QueryProcessingEvaluator Evaluates query understanding and processing quality. Assesses clarity, intent detection, preprocessing effectiveness, specificity, complexity, and ambiguity of input queries with configurable weights.
rag rule-based trace-evaluation +3
Threshold: 7 RerankingEvaluator Evaluates reranking stage effectiveness.
pipeline
Click for details Threshold: 7
ResourceUtilizationScorer Analyzes resource utilization across pipeline stages.
pipeline
Click for details Threshold: 7
RetrievalStageEvaluator Enhanced retrieval stage performance evaluation combining precision, recall, ranking, similarity, and diversity metrics. Provides comprehensive assessment of the retrieval component.
rag rule-based trace-evaluation +3
Threshold: 7 AnswerRefusalScorer Evaluates whether the model appropriately refuses to answer harmful or inappropriate requests. Assesses if the model correctly identifies dangerous queries and provides appropriate refusals. Important for compliance and guardrails.
safety conversational llm-judge +4
Threshold: 7 ContentModerationScorer Evaluates content against moderation guidelines and policies. Checks for violations across multiple categories including violence, hate speech, adult content, and self-harm. Essential for platform safety.
safety llm-judge trace-evaluation +3
Threshold: 7 ContentSafetyViolationScorer Detects specific safety policy violations in generated content. More granular than general moderation, focusing on specific prohibited content types with detailed violation reports and severity levels.
safety llm-judge trace-evaluation +3
Threshold: 7 IsHarmfulAdviceScorer Evaluates whether generated content contains harmful advice or dangerous recommendations. Detects advice that could cause physical, financial, legal, or psychological harm to users.
safety llm-judge trace-evaluation +3
Threshold: 7 ToxicityScorer Measures the level of toxic content in generated text. Evaluates text for various forms of toxicity including insults, threats, profanity, and harassment with specific toxic element identification.
safety bias llm-judge +4
Threshold: 7 CulturalSensitivityScorer Assesses cultural sensitivity and appropriateness of generated content. Evaluates whether text respects cultural differences, avoids cultural appropriation, and maintains sensitivity toward diverse cultural practices and beliefs.
bias safety llm-judge +4
Threshold: 7 NoAgeBiasScorer Evaluates content for age-based discrimination or stereotypes. Identifies ageist language, assumptions about capabilities based on age, or stereotypical portrayals of different age groups.
bias safety llm-judge +4
Threshold: 8 NoGenderBiasScorer Evaluates whether the generated content contains gender-based biases or stereotypes. Analyzes text for implicit or explicit gender discrimination, stereotypical portrayals, or language that unfairly associates traits or roles with specific genders.
bias safety llm-judge +4
Threshold: 8 NoRacialBiasScorer Detects racial or ethnic biases in generated content. Uses an LLM to identify stereotypes, discriminatory language, or unfair associations based on race or ethnicity. Examines both explicit statements and subtle linguistic patterns.
bias safety llm-judge +4
Threshold: 8 AccuracyScorer Calculates accuracy based on substring or token-level matching between prediction and ground truth. More lenient than ExactMatchScorer, allowing for partial matches.
accuracy rule-based benchmark +2
Click for details Threshold: 7
ExactMatchScorer Evaluates whether the prediction exactly matches the ground truth. Strictest form of accuracy measurement for tasks requiring exact output like code generation or classification labels.
accuracy rule-based benchmark +2
Click for details Threshold: 10
F1Scorer Computes token-level F1 score between prediction and ground truth. Balances precision and recall, useful for extractive tasks and named entity recognition.
accuracy nlp-metrics rule-based +4
Click for details Threshold: 7
MultiPatternAccuracyScorer Evaluates prediction accuracy against multiple acceptable patterns or answers. Ideal for tasks where multiple correct answers exist, with regex support for flexible pattern matching.
accuracy rule-based benchmark +3
Click for details Threshold: 7
BLEUScorer Computes BLEU (Bilingual Evaluation Understudy) score between prediction and ground truth. Measures n-gram precision with a brevity penalty, useful for machine translation and text summarization evaluation.
nlp-metrics accuracy rule-based +2
Click for details Threshold: 7
LevenshteinSimilarityScorer Computes similarity based on Levenshtein (edit) distance between prediction and ground truth. Measures the minimum number of single-character edits needed to transform one string into another.
nlp-metrics accuracy rule-based +2
Click for details Threshold: 7
ROUGEScorer Computes ROUGE scores for text summarization evaluation. Calculates ROUGE-1 (unigram), ROUGE-2 (bigram), and ROUGE-L (longest common subsequence) metrics for comprehensive content overlap analysis.
nlp-metrics accuracy rule-based +2
Click for details Threshold: 7
IsEmailScorer Validates whether the prediction contains a valid email address format. Uses regular expression matching to check for properly formatted email addresses in standalone strings or text.
format rule-based trace-evaluation +2
Threshold: 10 IsJSONScorer Validates whether the prediction is valid JSON. This rule-based scorer attempts to parse the prediction as JSON and returns a perfect score if parsing succeeds. Useful for structured output tasks.
format agent rule-based +3
Threshold: 10 JSONSchemaScorer Validates that JSON output conforms to a specified JSON Schema. Checks JSON validity and ensures the structure matches expected fields, types, and constraints defined in a provided schema.
format agent rule-based +4
Threshold: 10 ValidLinksScorer Validates that URLs in the prediction are properly formatted and optionally accessible. Extracts URLs from text, validates their format, and can optionally check if they resolve to valid resources.
format rule-based trace-evaluation +2
Threshold: 10 