Lack of Visibility
It was impossible to pinpoint why an agent failed from terabytes of logs.
Customer storyAI Voice Agents • Enterprise SaaS
MyOperator Achieved 4-6x AI Performance with Evidence-Led Optimization
How India's leading call center platform used production traces, configured evaluations, and engineering-reviewed recommendations to improve its AI agents.
Built for Enterprise4-line SDK integrationResults in < 24 hours
Verified outcomesOverall Success Rate84%95%+
Context Relevancy1.47/108.5/10
Role Adherence1.99/108.5/10
4 Lines of Code200xFaster Optimization
India's #1 Cloud Call Center Platform
MyOperator is India's #1 cloud-based call management system, trusted by 12,000+ businesses to power their customer communications with AI.
Cloud Call CenterAI Voice AgentsEnterprise SaaS
01: Customer
India's Leading Business Communication Platform
Founded in 2013, MyOperator has grown to become the go-to platform for enterprise-grade AI-powered communication solutions. Their intelligent voice bots and AI agents handle millions of customer interactions daily, making them a perfect candidate for advanced AI observability.
12,000+
Enterprise Customers
100+
AI Voice Agents
50K+
Daily Calls Processed
Millions+
Monthly AI Interactions
02: The problem
The Challenge: Flying Blind at Scale
MyOperator's AI agents were a core part of their business, but the engineering team faced a critical challenge: they were flying blind. With a massive volume of interactions, they had no efficient way to diagnose the root causes of agent failures.
Unscalable Manual Effort
Manually reviewing conversations and testing fixes was slow, inefficient, and couldn't keep up with the volume.
Inability to Catch Subtle Failures
Nuanced issues like agent hallucinations or persona drift were nearly impossible for human engineers to detect at scale.
“Our customers would report an issue, and we'd be lost. We had the data, but we couldn't turn it into answers. We were spending more time searching for problems than solving them.”
03: What we did
Under the Hood: The Evaluation and Review Workflow
Noveum connected production traces, structured datasets, configured evaluations, NovaPilot reports, and engineering review in one evidence-led workflow.
Scheduled reports - engineering review requiredRe-evaluate after implementation
1
Real-Time Trace Collection
2
Automated ETL & Dataset Enrichment
3
Configured Evaluation with 20+ Scorers
4
NovaPilot Pattern Analysis & Recommendations
5
Engineering Review & Implementation
6
Team Validation & Deployment
Real-Time Trace Collection
After a simple 4-line-of-code integration with MyOperator's LangChain setup, the Noveum platform began collecting instrumented production traces for analysis.
Automated ETL & Dataset Enrichment
A sophisticated ETL (Extract, Transform, Load) job runs continuously, transforming the raw, unstructured traces into clean, structured dataset items. This process enriches the data, making it ready for deep analysis.
Configured Evaluation with 20+ Scorers
Structured dataset items were evaluated with more than 20 selected scorers, covering criteria such as factual accuracy and role adherence.
NovaPilot Pattern Analysis & Recommendations
NovaPilot analyzed the configured evaluation results, grouped recurring failures, and produced an evidence-linked report with issue patterns and recommended next steps.
Engineering Review & Implementation
Engineers reviewed NovaPilot's ranked recommendations against the linked items and traces, then chose which changes to implement.
Team Validation & Deployment
The team tested implemented changes, compared new evaluation results with the baseline, and retained only changes it had reviewed and verified before deployment.
This workflow shortened the path from a production failure to evidence, a prioritized recommendation, an engineering decision, and a new evaluation.
04 - How it analyzes
From Evidence to a Reviewed Change
NovaPilot organizes evaluated failures and recommends changes. Engineers decide what to implement, test the result, and keep human judgment in the release path.
01
Diagnose
Group issues with linked evidence
02
Recommend
Rank evidence-linked recommendations
03
Implement
Engineers choose and make the change
04
Validate
Re-run evaluations before release
Engineering Review Stages
Baseline75 discarded
100
Recommendation17 discarded
25
Implementation5 discarded
8
Re-evaluation2 discarded
3
Team-validated change - ready for a release decision
Without Structured Evidence
- Engineers search raw logs manually
- Failures are difficult to group
- Changes lack a shared baseline
- Review context is scattered
With NovaPilot Reports
- Issues are grouped with source evidence
- Recommendations are ranked for review
- Evaluation results provide a baseline
- Engineers retain the release decision
05 - The recommendations
Evidence-Backed Recommendations
NovaPilot surfaced prompt recommendations from the evaluation evidence. The MyOperator team reviewed, implemented, and evaluated these five changes, with the measured results shown below:
136+
Recommendation Candidates Reviewed
4
Review Iterations
5
Team-Validated Changes
10min
Minutes to Analyze
explicit_domain_scope_constraint - evidence-linked recommendationDelivered as an analysis report
An example change the team reviewed
Before
"domain_scope": "You are ONLY authorized to answer questions directly related to Heyo products..."
After - team-verified change
"domain_scope": "You are ONLY authorized to answer questions directly related to Heyo products... For all other topics, respond politely that you can only assist with Heyo-related queries."
Top 5 Team-Validated Changes
Reducing Critical Hallucinations
explicit_domain_scope_constraintProblem detected
The agent was inventing features and omitting critical data, causing users to receive incorrect information about product capabilities.
NovaPilot recommendation
NovaPilot recommended a strict domain scope constraint; the team implemented it, defined clear boundaries, and evaluated the result.
Hallucination Detection+6.7
2.8/10
9.5/10
Role Adherence+6.7
2.5/10
9.2/10
Strengthening Agent Identity
simplify_and_focus_identity_statementProblem detected
The agent was confused about its own identity and persona, leading to inconsistent and confusing responses.
NovaPilot recommendation
NovaPilot recommended a simplified identity statement; the team implemented and evaluated the clearer role and personality definition.
Role Adherence+7.0
2.2/10
9.2/10
Q&A Alignment+8.4
1.1/10
9.5/10
Clarifying SaaS Context
clarify_heyo_context_as_saas_platformProblem detected
The agent didn't understand it was representing a SaaS platform, leading to incorrect assumptions about product delivery and support.
NovaPilot recommendation
NovaPilot recommended explicit SaaS-platform context; the team added it and evaluated the resulting behavior.
Role Adherence+7.0
2.2/10
9.2/10
Q&A Alignment+8.5
1.0/10
9.5/10
Amplifying Domain Constraints
amplify_domain_constraint_with_examplesProblem detected
The agent was still occasionally answering out-of-scope questions, especially edge cases not covered by the original constraints.
NovaPilot recommendation
NovaPilot recommended concrete in-scope and out-of-scope examples; the team added them and evaluated the clearer boundaries.
Hallucination Detection+5.7
3.8/10
9.5/10
Role Adherence+6.5
3.0/10
9.5/10
Enforcing Truthfulness
truth_first_constraint_and_unknown_responseProblem detected
The agent would sometimes guess instead of admitting it didn't know, leading to confident but incorrect answers.
NovaPilot recommendation
NovaPilot recommended a truth-first constraint and unknown-response template; the team implemented and evaluated the change.
Q&A Alignment+8.6
0.9/10
9.5/10
Role Adherence+6.7
2.5/10
9.2/10
Additional Behavioral Recommendations
Beyond the core findings, NovaPilot identified these behavioral issues and recommended changes for the team to review:
The agent was making up its own follow-up questions, leading to inconsistent user experiences.
Recommendation: add a rule that forbids the agent from improvising questions.
The agent was confused about its own identity, sometimes referring to itself as a 'bot' or 'agent.'
Recommendation: clarify the persona rule and its required identity behavior.
The agent would try to access user account details, which it couldn't do, leading to a frustrating dead-end for the user.
Recommendation: state that account status cannot be checked and provide a clear escalation path.
Users with complex issues had no clear path to human support, causing frustration and abandoned conversations.
Recommendation: define a reviewed protocol for offering human handoff when frustration signals appear.
What Didn't Work: Failed Experiments
Not every proposed change improved the measured results. The team used these failed experiments as review evidence:
remove_hedging_from_domain_scopeRemoved helpful examples from the prompt, intending to make it more direct and concise.
-4.7 score drop
add_verbose_context_explanationAdded lengthy explanations about the product to help the agent understand context better.
-2.3 score drop
strict_one_sentence_responsesForced all responses to be single sentences for brevity.
-3.1 score drop
Why Failures Matter
These failures were valuable evidence. The engineering team rejected changes that hurt performance and retained control over what reached production.
06: Verified impact
Verified Performance and a Paradigm Shift
The cumulative effect of the team-reviewed and evaluated changes produced the following measured performance gains:
Context Relevancy5.8x
1.47/10
8.5/10
Role Adherence4.3x
1.99/10
8.5/10
Overall Success Rate+11%
84%
95%+
The Business Transformation
20-30%
Reduction in AI Engineering Overhead
Freed up expensive engineering resources from manual debugging.
200x
Faster Workflow
Shortened the feedback loop from weeks to minutes.
∞
Scale with Confidence
Enabled the team to manage hundreds of agents without a linear increase in team size.
07: Why Noveum
The Complete Solution
MyOperator chose Noveum.ai to connect instrumented traces, configured evaluation, and evidence-led NovaPilot recommendations in one review workflow.
Evidence-Led Review Workflow
NovaPilot reports kept issue patterns, recommendations, and evaluated source items together while engineers retained implementation and release decisions.
Verifiable, Data-Driven Results
Configured evaluations gave the team a measurable baseline for comparing changes it implemented and reviewed.
Unprecedented Speed and Agility
Faster analysis and organized evidence helped the team shorten its engineering review and evaluation loop.
Conclusion
The partnership between MyOperator and Noveum.ai proves a fundamental truth of the modern AI stack: you cannot afford to fly blind. The challenge of ensuring quality at scale is a machine-scale problem that requires a machine-scale solution.
Noveum combines fast tracing integration, configured evaluation, and NovaPilot reports so teams can investigate failures, review recommendations, and verify their own changes with evidence.
Ready to Transform Your AI Operations?
Stop Flying Blind
See how Noveum.ai can organize tracing, evaluation, and recommendation evidence for your AI quality workflow.