NovaEval - AI Evaluation Engine

NovaEval is Noveum's comprehensive AI evaluation engine. It provides an intelligent, automated interface for evaluating AI agents, voice assistants, RAG pipelines, and conversational systems — all integrated seamlessly into your Noveum dashboard.

About NovaEval

NovaEval powers Noveum's evaluation pipeline with 80+ specialized scorers across 10 categories. It automatically selects the most relevant scorers based on your dataset's content and presents results directly in your Noveum dashboard.

Key Capabilities

• Comprehensive Scorer Library: 80+ specialized scorers covering accuracy, RAG, conversational, agent, audio quality, latency, telephony, safety, LLM-as-judge, and custom domains
• Voice Agent Evaluation: Audio quality scorers (MOS, pronunciation, gibberish detection) and latency scorers (TTFT, STT, TTS, E2E) built specifically for voice agents
• Telephony Scorers: Purpose-built scorers for phone agents — instruction adherence, CSAT, drop-off node detection, call termination quality
• Intelligent Selection: "Recommend Scorers" analyzes your dataset and suggests the best scorers automatically
• Custom Scorer Support: Create domain-specific evaluation metrics when standard scorers don't meet your needs
• Batch and Realtime modes: Score historical datasets or continuously score new items as they arrive
• Production Ready: Built for enterprise-scale evaluation with robust error handling and scalability

Why Model Evaluation Matters

AI models can perform differently across various scenarios, and understanding their strengths and weaknesses is crucial for:

• Performance Optimization: Identify which models work best for specific use cases
• Cost Efficiency: Find the most cost-effective models without sacrificing quality
• Quality Assurance: Ensure consistent, reliable outputs across different scenarios
• Continuous Improvement: Track model performance over time and identify degradation

Comprehensive Scoring Framework

NovaEval provides a comprehensive suite of scorers organized by evaluation domain. Most scorers follow a shared scorer interface and support both synchronous and asynchronous evaluation. Agent-specific evaluators use dedicated evaluation primitives and are intentionally standalone.

🎯 Accuracy & Classification Metrics

📊 NLP Metrics

💬 Conversational AI Metrics

🔍 RAG (Retrieval-Augmented Generation) Metrics

📝 Answer Quality Metrics

🛡️ Hallucination Detection Metrics

🔀 Multi-Context Metrics

🔒 Safety & Moderation Metrics

⚖️ Bias Detection Metrics

✅ Format Validation Metrics

🤖 LLM-as-Judge Metrics

🎭 Agent Evaluation Metrics

🔧 RAG Pipeline Metrics

How Noveum Handles Evaluation

Noveum automatically handles the entire evaluation process for you, making it seamless and effortless:

1. Automatic Trace Processing

• Your AI traces are automatically processed and converted into evaluation datasets
• No manual data preparation or configuration required
• Real-world data from your actual AI interactions

2. Intelligent Scorer Selection

Based on your specific use case and AI application type, Noveum automatically selects the most relevant scorers from our comprehensive suite:

• RAG Applications: Contextual precision, faithfulness, answer relevancy
• Conversational AI: Knowledge retention, conversation relevancy, role adherence
• Agent Systems: Tool correctness, task progression, goal achievement
• Classification Tasks: Accuracy, F1 scores, exact match evaluation
• Voice Agents: MOS score, tone clarity, pronunciation, gibberish detection, E2E latency, STT/TTS latency
• Telephony / Phone: Instruction adherence, CSAT sentiment, drop-off node detection, appropriate call termination

3. Custom Scorer Creation

When standard scorers don't meet your specific requirements, Noveum enables you to:

• Create custom evaluation metrics tailored to your business needs
• Define domain-specific scoring criteria
• Implement proprietary evaluation logic
• Integrate with your existing evaluation frameworks

4. Dashboard Integration

All evaluation results are automatically presented in your Noveum dashboard:

• Real-time scoring and performance metrics
• Comparative analysis across different models
• Historical performance tracking
• Actionable insights and recommendations

Choosing the Right Scorers

With our extensive collection of 80+ specialized scorers, selecting the right combination is crucial for meaningful evaluation:

For RAG Applications

• AnswerRelevancyScorer: Ensures answers directly address the questions
• FaithfulnessScorer: Prevents hallucinations and maintains factual accuracy
• ContextualPrecisionScorer: Validates relevance of retrieved context
• ContextualRecallScorer: Ensures comprehensive information coverage

For Conversational AI

• KnowledgeRetentionScorer: Tracks information retention across conversations
• ConversationRelevancyScorer: Maintains context-aware responses
• RoleAdherenceScorer: Ensures consistent persona maintenance
• ConversationCompletenessScorer: Validates complete request fulfillment

For Agent Systems

• Tool Relevancy Scoring: Validates appropriate tool selection
• Task Progression Scoring: Measures goal advancement
• Goal Achievement Scoring: End-to-end task completion assessment
• Context Relevancy Scoring: Role-task alignment validation

For Voice Agents (LiveKit / Pipecat / NovaSynth)

• MOS Scorer: Mean Opinion Score — overall perceived audio quality (0–10 in dashboard)
• Tone Clarity Scorer: Naturalness and clarity of synthesized speech
• Gibberish Scorer: Detects garbled or unintelligible audio segments
• Pronunciation Audio Scorer: Pronunciation accuracy across the session
• E2E Latency Scorer: End-to-end latency from user speech end to agent response start
• LLM TTFT Scorer: LLM time-to-first-token
• TTS TTFB Scorer: TTS time-to-first-byte

For Telephony / Phone Agents

• Instruction Adherence Scorer: Whether the agent followed its instructions throughout the call
• Sentiment CSAT Scorer: Estimated customer satisfaction score based on conversation sentiment
• Drop-off Node Scorer: Identifies where in the conversation the user disengaged
• Conversation Context Coherence Scorer: Coherence across the full call
• Appropriate Call Termination Scorer: Whether the call ended appropriately

For Custom Requirements

When standard scorers don't meet your needs, create custom evaluation metrics that align with your specific business objectives and domain requirements.

Next Steps

• Scorers Reference — complete reference for all 80+ scorers with required fields
• Running Evaluations — create an eval job step by step
• SDK Integration — set up tracing to capture data for evaluation
• NovaSynth — generate voice test data for audio/telephony scorers
• NovaPilot — AI analysis and recommendations on eval results