Lack of Visibility
It was impossible to pinpoint why an agent failed from terabytes of logs.
Customer storyAI Voice Agents • Enterprise SaaS
MyOperator Achieved 4-6x AI Performance with Autonomous Optimization
How India's leading call center platform transformed their AI agents from unreliable to production-ready using Noveum's 24/7 autonomous optimization engine.
Built for Enterprise4-line SDK integrationResults in < 24 hours
Verified outcomesOverall Success Rate84%95%+
Context Relevancy1.47/108.5/10
Role Adherence1.99/108.5/10
4 Lines of Code200xFaster Optimization
India's #1 Cloud Call Center Platform
MyOperator is India's #1 cloud-based call management system, trusted by 12,000+ businesses to power their customer communications with AI.
Cloud Call CenterAI Voice AgentsEnterprise SaaS
01 — Customer
India's Leading Business Communication Platform
Founded in 2013, MyOperator has grown to become the go-to platform for enterprise-grade AI-powered communication solutions. Their intelligent voice bots and AI agents handle millions of customer interactions daily, making them a perfect candidate for advanced AI observability.
12,000+
Enterprise Customers
100+
AI Voice Agents
50K+
Daily Calls Processed
Millions+
Monthly AI Interactions
02 — The problem
The Challenge: Flying Blind at Scale
MyOperator's AI agents were a core part of their business, but the engineering team faced a critical challenge: they were flying blind. With a massive volume of interactions, they had no efficient way to diagnose the root causes of agent failures.
Unscalable Manual Effort
Manually reviewing conversations and testing fixes was slow, inefficient, and couldn't keep up with the volume.
Inability to Catch Subtle Failures
Nuanced issues like agent hallucinations or persona drift were nearly impossible for human engineers to detect at scale.
“Our customers would report an issue, and we'd be lost. We had the data, but we couldn't turn it into answers. We were spending more time searching for problems than solving them.”
03 — What we did
Under the Hood: The 24/7 Continuous Optimization Pipeline
Noveum.ai's platform is more than a tool; it's a perpetual, autonomous quality assurance and optimization engine that runs 24/7.
Runs 24/7 — no human in the loopLoops continuously
1
Real-Time Trace Collection
2
Automated ETL & Dataset Enrichment
3
Continuous Evaluation with 20+ Scorers
4
NovaPilot Analysis & Root Cause Identification
5
Evolutionary Auto-Fix & Simulation
6
Delivery of Verified Fixes
Real-Time Trace Collection
After a simple 4-line-of-code integration with MyOperator's LangChain setup, the Noveum.ai platform began to passively collect raw production traces from every agent interaction, without impacting performance.
Automated ETL & Dataset Enrichment
A sophisticated ETL (Extract, Transform, Load) job runs continuously, transforming the raw, unstructured traces into clean, structured dataset items. This process enriches the data, making it ready for deep analysis.
Continuous Evaluation with 20+ Scorers
As new dataset items are created, they are immediately evaluated against Noveum.ai's suite of 20+ specialized scorers. These scorers, covering everything from Factual Accuracy to Role Adherence, provide a multi-dimensional, quantitative assessment of every single interaction.
NovaPilot Analysis & Root Cause Identification
NovaPilot, Noveum.ai's own agent, analyzes the firehose of scored data in real-time. It identifies patterns, clusters failures, and performs a root cause analysis to pinpoint the underlying issues that are causing poor performance.
Evolutionary Auto-Fix & Simulation
Based on its analysis, NovaPilot generates hundreds of potential fixes (e.g., system prompt modifications). It then initiates an evolutionary optimization process, running simulations to test these fixes across multiple generations.
Delivery of Verified Fixes
The 'winning' prompt—the one that is mathematically proven to deliver the highest performance—is then delivered to the engineering team, ready for deployment. This entire cycle, from collecting a trace to having a verified fix, runs continuously, 24/7, without any human intervention.
This 'Set It and Forget It' pipeline transformed MyOperator's reactive, manual process into a proactive, autonomous, and perpetual improvement loop.
04 — How it optimizes
Evolutionary Optimization: Natural Selection for AI Prompts
Like biological evolution, NovaPilot tests hundreds of potential fixes across multiple generations. The best performers survive and combine, while poor performers are discarded—delivering mathematically proven improvements.
01
Generate
Create 100+ prompt variations
02
Simulate
Test each against real data
03
Evaluate
Score with 20+ metrics
04
Evolve
Keep winners, combine traits
Multi-Generation Selection Process
Generation 175 discarded
100
Generation 217 discarded
25
Generation 35 discarded
8
Final2 discarded
3
Verified Winner Emerges — Ready for Production
Without Evolutionary Optimization
- Manual A/B testing takes weeks
- Only 2-3 variations tested at a time
- No guarantee of optimal solution
- Human bias influences prompt selection
With NovaPilot's Evolutionary Engine
- Automated testing completes in minutes
- 100+ variations tested per generation
- Mathematically proven optimal prompt
- Data-driven selection, no human bias
05 — The fixes
Complete List of AI-Generated Fixes
NovaPilot tested hundreds of prompt variations across 4 generations. Here are the top 5 verified fixes that delivered the most significant improvements, along with their measured impact:
136+
Prompt Variations Tested
4
Evolution Generations
5
Verified Winning Fixes
10min
Minutes to Optimize
explicit_domain_scope_constraint — generation 4 winnerDelivered as a pull request
The exact change NovaPilot shipped
Before
"domain_scope": "You are ONLY authorized to answer questions directly related to Heyo products..."
After — verified fix
"domain_scope": "You are ONLY authorized to answer questions directly related to Heyo products... For all other topics, respond politely that you can only assist with Heyo-related queries."
Top 5 Verified Winning Fixes
Fixing Critical Hallucinations
explicit_domain_scope_constraintProblem detected
The agent was inventing features and omitting critical data, causing users to receive incorrect information about product capabilities.
NovaPilot's fix
Introduced a strict domain scope constraint that explicitly defines boundaries and politely redirects out-of-scope questions.
Hallucination Detection+6.7
2.8/10
9.5/10
Role Adherence+6.7
2.5/10
9.2/10
Strengthening Agent Identity
simplify_and_focus_identity_statementProblem detected
The agent was confused about its own identity and persona, leading to inconsistent and confusing responses.
NovaPilot's fix
Implemented a simplified, focused identity statement that clearly defines the agent's role and personality.
Role Adherence+7.0
2.2/10
9.2/10
Q&A Alignment+8.4
1.1/10
9.5/10
Clarifying SaaS Context
clarify_heyo_context_as_saas_platformProblem detected
The agent didn't understand it was representing a SaaS platform, leading to incorrect assumptions about product delivery and support.
NovaPilot's fix
Added explicit SaaS platform context, clarifying that Heyo is a cloud-based service with online support and self-service features.
Role Adherence+7.0
2.2/10
9.2/10
Q&A Alignment+8.5
1.0/10
9.5/10
Amplifying Domain Constraints
amplify_domain_constraint_with_examplesProblem detected
The agent was still occasionally answering out-of-scope questions, especially edge cases not covered by the original constraints.
NovaPilot's fix
Added concrete examples of in-scope vs out-of-scope questions to the domain constraint, making boundaries crystal clear.
Hallucination Detection+5.7
3.8/10
9.5/10
Role Adherence+6.5
3.0/10
9.5/10
Enforcing Truthfulness
truth_first_constraint_and_unknown_responseProblem detected
The agent would sometimes guess instead of admitting it didn't know, leading to confident but incorrect answers.
NovaPilot's fix
Implemented a 'truth-first' constraint that prioritizes accuracy over helpfulness, with a graceful unknown response template.
Q&A Alignment+8.6
0.9/10
9.5/10
Role Adherence+6.7
2.5/10
9.2/10
Additional Behavioral Fixes
Beyond the core optimizations, NovaPilot also identified and fixed these behavioral issues:
The agent was making up its own follow-up questions, leading to inconsistent user experiences.
Added a new critical rule: THE 'NO IMPROV' RULE (CRITICAL), strictly forbidding the agent from improvising questions.
The agent was confused about its own identity, sometimes referring to itself as a 'bot' or 'agent.'
Strengthened the persona rule: You must never reveal you are a bot. You are a human specialist named Riva.
The agent would try to access user account details, which it couldn't do, leading to a frustrating dead-end for the user.
Added a NO-API guardrail, explicitly stating that it cannot check a user's status and providing a clear escalation path.
Users with complex issues had no clear path to human support, causing frustration and abandoned conversations.
Implemented a smart escalation protocol that detects frustration signals and proactively offers human agent handoff.
What Didn't Work: Failed Experiments
Not every AI-generated fix succeeded. Here's what NovaPilot learned from its failures:
remove_hedging_from_domain_scopeRemoved helpful examples from the prompt, intending to make it more direct and concise.
-4.7 score drop
add_verbose_context_explanationAdded lengthy explanations about the product to help the agent understand context better.
-2.3 score drop
strict_one_sentence_responsesForced all responses to be single sentences for brevity.
-3.1 score drop
Why Failures Matter
These failures are actually valuable—they prove the evolutionary process works. NovaPilot automatically discards changes that hurt performance, ensuring only verified improvements make it to production.
06 — Verified impact
Verified Performance and a Paradigm Shift
The cumulative effect of all the AI-generated improvements resulted in a massive overall performance gain:
Context Relevancy5.8x
1.47/10
8.5/10
Role Adherence4.3x
1.99/10
8.5/10
Overall Success Rate+11%
84%
95%+
The Business Transformation
20-30%
Reduction in AI Engineering Overhead
Freed up expensive engineering resources from manual debugging.
200x
Faster Workflow
Shortened the feedback loop from weeks to minutes.
∞
Scale with Confidence
Enabled the team to manage hundreds of agents without a linear increase in team size.
07 — Why Noveum
The Complete Solution
MyOperator chose Noveum.ai because it offered a complete, end-to-end solution that went beyond simple monitoring.
Fully Autonomous Engine
NovaPilot handled the entire quality assurance lifecycle, 24/7, without human intervention.
Verifiable, Data-Driven Results
The evolutionary process didn't just suggest fixes; it proved them with data through rigorous simulation.
Unprecedented Speed and Agility
The 10-minute optimization cycle was a game-changer, enabling rapid iteration and continuous improvement.
Conclusion
The partnership between MyOperator and Noveum.ai proves a fundamental truth of the modern AI stack: you cannot afford to fly blind. The challenge of ensuring quality at scale is a machine-scale problem that requires a machine-scale solution.
Noveum.ai, with its 15-minute integration and the 24/7 autonomous, evolutionary capabilities of NovaPilot, provides that solution. It is an essential platform for any enterprise that is serious about deploying reliable, high-performing AI agents.
Ready to Transform Your AI Operations?
Stop Flying Blind
See how Noveum.ai can help you automate your AI quality assurance and unlock the full potential of your agents.