A Specification for Voice AI Evaluation

TL;DR: Most voice AI apps are not doing evaluations because several things matter in a real conversations: timing, interruptions, and task completion. This post introduces a practical specification for evaluating voice AI platforms using synthetic data, with clear metrics for latency, flow, and recovery. It’s designed for teams building or buying production-ready voice systems.

Why a Specification?

Most teams evaluate voice AI with ad-hoc tests that miss key conversation behaviours. Across industries, I’ve seen the same gaps: How do you measure interruption handling? What’s an acceptable latency? How do you tell if a bot sounds natural?

This is not a think-piece it is an initial specification. Whether you’re choosing a platform like Hamming, Coval, Freestyle, or Arise, or building from scratch, this evolving framework defines comprehensive testing. Contributions welcome, DM me on Twitter @vr000m.

Specifications force clarity. Each requirement serves a purpose. Each metric has a target. Use the entire framework or just what fits. It provides a shared language for evaluating voice AI quality.

Voice AI Evaluation Specification v0.1

Changelog:

v0.1 (30 July) – Initial release of evaluation criteria and test design for voice AI systems

1. Purpose & Scope

This specification sets out how to evaluate voice AI systems in multi-turn conversations. It focuses on measuring performance, interaction quality, and control—ensuring systems behave well in real-world settings.

The Challenge of Non-Determinism

Voice AI systems combine multiple non-deterministic components: LLMs generate different responses to identical prompts, VAD triggers vary with minor audio variations, and STT confidence scores fluctuate. Because of this variability, a single test is meaningless. Repeated testing provides statistical confidence. Temperature settings alone can transform a concise assistant into a chatty companion. This is why continuous evaluation is not optional—it’s essential.

Why Synthetic Data Matters

Using real customer conversations for testing creates three problems:

1. Privacy compliance: GDPR, CCPA, and HIPAA make using real conversations legally complex
2. Reproducibility: You can not debug intermittent issues without consistent test inputs
3. Edge case coverage: Real data may not yet include all the edge cases that break systems

Synthetic data enables regression testing. When the LLM changes or prompts are adjusted, you can measure the impact immediately.

Setting Expectations

This specification covers system-level evaluation, not model training or prompt optimization. It answers questions like:

• Does my complete voice AI system meet latency requirements?
• How gracefully does it handle interruptions and errors?
• Will it perform consistently across diverse user populations?

It does not cover:

• How to train or fine-tune language models
• Acoustic model optimization
• Infrastructure scaling strategies

Integration in Your Development Lifecycle

Successful teams integrate voice AI evaluation at three stages:

1. Pre-deployment testing: Run the full test suite before any production release
2. A/B testing: Compare configurations and measure outcomes that have statistical significance
3. Production monitoring: Sample real conversations against your baseline metrics

Automation is key. Tests should run like unit tests—on commits or schedules. A dashboard showing overnight performance drift across your test suite is invaluable for catching model updates, configuration changes, or emergent behaviours before customers notice. This results in the following core principles:

• All evaluation must use synthetic data to ensure reproducibility
• Tests must cover both technical performance and conversational dynamics
• Evaluations should be automated and CI/CD compatible
• Results must be comparable across different configurations
• Routine testing is essential—LLM variability demands daily or per-change runs

2. Use Case Coverage

Your evaluation framework should support a wide range of conversational patterns across industries. Testing requirements depend on the application or use-case.

Transactional Flows

Example: Pizza ordering bot

User: "I want a large pepperoni pizza"
Bot: "One large pepperoni pizza. Would you like to add any drinks?"
User: "Actually make that two pizzas and add a coke"
Bot: "Updated to two large pepperoni pizzas and one Coke. Anything else?"

Key evaluation areas:

• Order updates and corrections
• Confirmation accuracy
• Correct total calculation
• Secure payment flow

Test how the bot handles mid-order changes like "wait, change that to…" and maintains state.

Information Gathering

Example: Healthcare intake bot

Bot: "What symptoms are you experiencing today?"
User: "I've had a headache for three days and some dizziness"
Bot: "I understand. On a scale of 1-10, how severe is your headache?"
User: "Maybe a 7... no actually it's more like an 8"

Key evaluation areas:

• Accurate data capture
• Effective clarification strategies
• Privacy-compliant handling

Use varied phrasing, from colloquial ("my head hurts") to clinical ("cephalgia and vertigo"), to test understanding.

Complex Navigation

Example: Banking IVR replacement

User: "I need to dispute a charge on my credit card"
Bot: "I can help with that. First, I'll need to verify your identity. Can you provide your account number?"
User: "I don't have it with me"
Bot: "No problem. I can look it up with your social security number and date of birth instead."

Key evaluation areas:

• Handling partial or alternate user information
• Robust fallback navigation
• Support for uncommon queries

Design tests that follow multiple paths, including informal input and edge cases.

Voicemail and IVR Tree Navigation

Example: Appointment reminder bot

Bot: "Hi, this is Dr. Smith's office calling to confirm your appointment tomorrow at 2 PM"
[Voicemail beep detected]
Bot: [Switches to voicemail mode] "This is a reminder about your appointment tomorrow at 2 PM. Please call us back at..."

Key evaluation areas:

• Voicemail and IVR detection
• Timely delivery of critical information

Include test cases for common and custom voicemail greetings, business IVRs, and delayed beep scenarios.

Escalation Paths

Example: Customer service bot with human handoff

User: "This is ridiculous, I've been trying to resolve this for a few minutes!"
Bot: "I understand your frustration. Let me connect you with a representative who can help immediately."
[Bot summarizes context for human agent]

Key evaluation areas:

• Accurate detection of frustration or complexity
• Clear escalation logic
• Quality of context summarisation

Test explicit handoff requests, tone-triggered escalations, and repeated failure cases.

Contextual Conversations

Example: Insurance claim bot

User: "I need to file a claim for my car accident"
Bot: "I'll help you with that. When did the accident occur?"
User: "Last Tuesday"
Bot: "That would be October 15th. Where did it happen?"
User: "The same intersection where I had that other claim last year"
Bot: "I see you had a claim at Main and 5th Street. Is that the location?"

Key evaluation areas:

• Reference resolution (time, place, previous interactions)
• Long-term memory or cross-session recall
• Clarification without user frustration

These scenarios test whether the bot can recall relevant information and resolve references naturally.

Your framework must support domain-specific priorities—e.g., 100ms latency may be critical for fast food but irrelevant for insurance claims. Design flexible scoring and thresholds tailored to each use case.

3. Data Requirements

3.1 Synthetic Test Data Generation

Effective synthetic data must cover the full acoustic and conversational range your system will face in production.

Voice Synthesis Setup

Build a baseline voice library with:

• Demographics: Diverse age groups and genders
• Regional accents: US, UK, Irish, Australian, Indian English, etc.
• Speaking patterns: Fast, slow, mumbling, clear, and casual speech
• Speech characteristics: Filler words, nervousness, varying articulation

Most TTS providers support voice and rate controls; simulate other traits via prompt engineering or audio processing.

Environmental Conditions

Add realistic audio degradation to clean speech:

• Background noise: Office, traffic, café, construction
• Network conditions: Packet loss (1–5%), jitter (10–100ms), compression artifacts
  • Device simulation: Mobile, Bluetooth headset, speakerphone echo
  • Call quality: PSTN noise, VoIP compression, cellular signal fade

Implementation Pipeline

Use prompts to systematically generate diverse failure cases. Automate and version-control your data generation. Requirements:

• Generate configurable numbers of test scenarios (typically 100–1000 per run)
• Apply voice diversity across the test set (target 80% profile coverage)
• Include ambiguous intents, context confusion, and varied emotional states
• Add environmental conditions systematically (noise, network, device)
• Output audio in standard formats (16–24kHz WAV)
• Store all relevant metadata and logs with audio for accurate result correlation

4. Functional Requirements

With synthetic test data in place, define what to measure in conversation. These requirements turn scenarios into measurable conversation dynamics.

4.1 Conversation Dynamics

Prioritize natural conversation flow, not just transcription accuracy, under real-world conditions. Focus evaluation on:

Turn-taking Analysis

Every conversation has implicit timing. Key metrics:

• Response timing: User speech end to bot speech start
• Interruption handling: Speed of bot response to interruptions
• Context preservation: Retains context after interruptions
• Recovery: Smooth handling of misunderstandings
• Natural flow: Pause duration, prosody, rhythm

Thresholds vary by use case—what’s responsive for support may feel rushed for therapy.

Test timing with scenarios like:

• Rapid-fire questions: Multiple queries in sequence
• Hesitant speakers: Disfluent or uncertain speech
• Overlapping speech: User talks before bot finishes
• Fast transitions: User starts immediately after bot
• Early barge-ins: Interruptions in first few bot words
• Simultaneous speech: Both speak at once (can reveal latency)

Barge-in Handling

Users expect instant recognition when interrupting. Tests should cover:

1. Interruption detection accuracy: Avoid false positives
2. Speech cessation speed: TTS stops promptly
3. Context recovery: Bot understands what was interrupted
4. Resume capability: Continues appropriately if needed

Backchannel Processing

Backchannels (“mm-hmm”, “right”, “okay”) keep conversations natural. Test:

• Encouragement: “uh-huh”, “go on”, “I see”
• Agreement: “yes”, “right”, “exactly”
• Confusion: “huh?”, “what?”, “sorry?”
• Impatience: “yeah yeah”, “okay but…”

Bots should not treat every backchannel as a full turn but should acknowledge engagement.

Silence Management

Silence handling depends on context:

Adjust thresholds by intent—longer pauses are fine in form-filling, but not in rapid order flows.

4.2 Latency & Responsiveness

Every stage in the voice pipeline adds delay. Measure end-to-end performance, not just individual components.

Key latency components:

• VAD triggering: Speech start/stop to detection
• STT processing: Audio to transcript
• LLM inference: Transcript to response
• TTS synthesis: Response to audio
• Audio streaming: Delivering audio to user

sequenceDiagram
    autonumber
    participant U as User
    participant Mic as Capture
    participant VAD as VAD
    participant STT as STT
    participant LLM as LLM
    participant TTS as TTS
    participant AO as Audio Output

    U->>Mic: user_speech_start
    Note over U,Mic: t0 = user_speech_start

    Mic->>VAD: audio frames
    VAD-->>Mic: vad_detection (start)
    Note over U,VAD: vad_start_trigger_duration

    U-->>Mic: user_speech_stop
    Mic->>VAD: trailing audio
    VAD-->>Mic: vad_detection (stop)
    Note over U,VAD: vad_stop_trigger_duration

    Mic->>STT: audio segment
    STT-->>STT: decode + finalize
    STT-->>LLM: transcript_complete (final)
    Note over VAD,STT: stt_processing_duration

    LLM-->>LLM: generate tokens
    LLM-->>TTS: first_token
    Note over STT,LLM: llm_first_token_latency

    LLM-->>TTS: response_complete
    Note over STT,LLM: llm_complete_response_latency

    %% TTS starts synthesizing as soon as it can (may be at first_token)
    LLM->>TTS: synthesis_start
    TTS-->>TTS: synthesize audio
    TTS-->>AO: audio_generation_complete
    Note over LLM,TTS: tts_synthesis_duration

    TTS-->>AO: first_audio_packet_output
    Note over TTS,AO: audio_streaming_start_latency

    AO-->>U: bot_audio_start
    Note over U,AO: end_to_end_total_duration

Test latency under:

• Peak traffic: High concurrent usage
• Network degradation: 0–5% packet loss
• Model switching: Different STT/LLM/TTS backends
• Longer context: Increased conversation history
• Ambiguous input: Disambiguation scenarios

Progressive Retry Mechanisms

Test escalation patterns that avoid user frustration:

1. First failure: Gentle clarification (“Could you repeat that?”)
2. Second: More specific help
3. Third: Offer alternatives
4. Escalation: Human handoff or alternate channel

A/B Testing Infrastructure

Automate scenario variation:

• Generate multiple test variations per base scenario
• Apply different voice profiles and environmental conditions
• Vary complexity (simple vs multi-step)
• Ensure enough cases for statistical significance

5. Evaluation Metrics

With test data and functional requirements defined, you need clear, quantifiable metrics to measure system performance. This section outlines essential metrics, quality assessment, and production monitoring.

5.1 Core Performance Metrics

Every voice AI system should track these key metrics:

Time to First Audio (TTFA)

TTFA is the end-to-end latency from when a user stops speaking to when the bot’s first audio response begins. Human conversation gaps are typically 200–300ms, but for voice AI:

• Cascade systems (STT→LLM→TTS): 800–1200ms is excellent, up to 1500ms is acceptable
• Speech-to-speech models: 600–900ms with optimized hosting
• Distributed hosting: Add 100–200ms for network overhead

Under 1 second feels responsive; 1–1.5 seconds is tolerable; over 2 seconds risks user frustration and interruption. Architecture choice impacts TTFA: cascades offer more control, speech-to-speech is faster but less transparent, and co-located hosting reduces latency at higher infra cost.

Voice Activity Detection (VAD) Accuracy

VAD errors cause:

• False positives (>5%): Bot interrupts users
• False negatives (>3%): Bot misses input

Aim for 95–97% accuracy in clean audio, 85–90% in noisy conditions. Below 90%, user experience suffers.

Barge-in Response Time

When users interrupt, bots must respond quickly. Target <200ms for barge-in handling to reduce abandonment, especially in critical scenarios like healthcare.

Task Completion Rate

Measures how often users achieve their goal:

• Customer service: 85–90%
• Sales qualification: 70–75%
• Appointment booking: 90–95%
• Technical troubleshooting: 60–70%

Track by intent. Simpler flows (e.g. pizza order) should see higher rates than complex, multi-step tasks.

Single-Turn vs. Multi-Turn Performance

Evaluate both:

• Single-turn: Intent recognition, response completeness, consistent latency
• Multi-turn: Context retention, efficient turns-to-completion (3–5 is good), logical progression, recovery from confusion

Track separately; some bots excel in one area but not the other. If average turns exceed 15 for any intent, users will likely disengage.

5.2 Quality Assessment

Raw metrics show what happened; quality assessment shows if the experience was good.

LLM-Based Quality Scoring

Use LLMs to score conversation transcripts on:

• Understanding: Did the bot interpret intent correctly?
• Helpfulness: Was the response useful?
• Naturalness: Did the exchange flow well?
• Efficiency: Was the conversation concise?

Prompt example:

Evaluate this conversation on a 1–5 scale for: UNDERSTANDING, HELPFULNESS, NATURALNESS, EFFICIENCY. For each, give a score and a brief justification.

UNDERSTANDING: Did the bot correctly interpret user intent?
- Consider: Misheard words, wrong intent classification, missed context

HELPFULNESS: Did the bot provide useful responses?
- Consider: Complete answers, relevant information, problem resolution

NATURALNESS: Did the conversation flow naturally?
- Consider: Appropriate responses, good timing, personality consistency

EFFICIENCY: Was the conversation appropriately concise?
- Consider: Unnecessary questions, repetition, verbose responses

Track distributions, not just averages. Consistent 3.5s beat wild swings between 5 and 2.

Human Review

Supplement LLM scoring with targeted human review:

• High-value or sensitive conversations
• Failed tasks
• Edge or emotional cases

Review 1–2% of volume, focusing on outliers.

Sentiment Tracking

Monitor sentiment shifts during conversations. A successful flow moves from neutral, through possible frustration, to positive resolution. Declining sentiment, even with task completion, signals issues.

5.3 Production Monitoring

Metrics and alerting in production are critical.

Dashboards

Track in real time (1-minute granularity):

• P50/P90/P99 latency
• Active conversations
• Error rates (STT, TTS, LLM)
• Escalation (handoff) triggers

Set alerts for:

• P90 latency >1.5× baseline
• Error rate >2% in 5 minutes
• Escalation >20% above baseline

Borrow from contact center KPIs:

• Containment rate: Resolved without human
• Average handle time
• First call resolution
• Customer effort (survey)

Model Drift Detection

Performance can degrade due to language shifts, seasonal changes, or new user expectations. Flag >5% drops from 30-day baselines. Retrain quarterly, but act on sudden drops.

Summary

Start with core metrics, add quality assessment as you grow, and build monitoring to catch problems before users do.

Moving Forward

No single evaluation framework fits every use case. This specification offers a flexible foundation—whether you’re evaluating HIPAA-sensitive healthcare bots or emotionally intelligent crisis assistants. Systematic testing beats ad-hoc guesswork.

As you evaluate platforms like Hamming, Arise, or Coval, use this specification to ask the right questions.

Ask these questions of any vendor or internal system:

– Can it test what matters for your use case?
– Does it expose the metrics you need?
– Is it CI/CD compatible?

Once you’ve established reliable evaluation for your current system, you’re ready to explore adaptive architectures—where evaluation complexity rises, but so does performance potential.

Beyond This Specification: Adaptive Architectures (added: 15th Aug)

This specification assumes a relatively static architecture where the same models handle all conversation turns. However, emerging patterns in voice AI suggest more sophisticated approaches that would require rethinking these evaluation criteria.

Adaptive Model Selection represents the next evolution in voice AI architecture. Instead of using the same model throughout a conversation, systems dynamically route requests based on conversation context:

• Light turns (greetings, confirmations): Route to fast, smaller models achieving <800ms latency
• Complex reasoning: Switch to larger models, accepting 1500-2000ms for accuracy
• Critical moments (medical, financial): Use best available models regardless of latency

This approach could reduce average latency by 30-40% while maintaining accuracy where it matters. However, evaluating such systems requires new metrics:

• Routing accuracy: Did the system select the appropriate model for each turn?
• Transition smoothness: Do model switches create noticeable personality shifts?
• Cost optimisation: What percentage of turns use expensive models?
• Degradation patterns: How does the system perform when preferred models are unavailable?

If you’re considering adaptive architectures, treat this specification as your baseline. Establish solid evaluation practices for single-model systems first, then layer on the additional complexity of multi-model orchestration. The fundamentals—measuring latency, tracking completion rates, assessing naturalness—remain essential regardless of architectural sophistication.

Glossary

VAD (Voice Activity Detection): A signal processing technique used to detect when a speaker starts and stops talking. It impacts when the system listens, responds, or cuts off speech.

STT (Speech-to-Text): The transcription engine that converts spoken audio into text. Accuracy depends on model quality, domain vocabulary, and audio conditions.

TTS (Text-to-Speech): The synthesis engine that converts generated text responses into spoken audio. Evaluated by clarity, prosody, latency, and adaptability.

LLM (Large Language Model): The generative model used to produce responses based on text input. LLM latency and variability affect conversation flow and tone.

TTFA (Time to First Audio): The time from the end of user speech to the beginning of the bot’s audio response. A key metric for conversational responsiveness.

Barge-in: When a user interrupts the bot mid-sentence. A good system detects this quickly, stops speaking, and adjusts its response contextually.

Containment Rate: Percentage of conversations resolved without human escalation. High containment indicates successful task completion by the bot.

Escalation: The process of handing a conversation off to a human agent or switching to a fallback system when the bot cannot proceed.

End-to-End Latency: Total time from the beginning of user speech to the start of bot speech, including VAD, STT, LLM, TTS, and streaming delays.