• October 19, 2026
• 10 min read

DeepEval Guide — Part 8b: Voice Evaluation¶

Voice agents fail in ways a transcript never shows: a clipped reply, a dead stream in the middle of a sentence, a 3-second silence before the agent answers, a voice that changes between turns. DeepEval 4.2 (August 2026) added voice evaluation: simulate a phone call against your agent, keep the audio, and score the call with metrics that work on the audio signal itself.

Version note. Voice evaluation appeared in deepeval 4.2.0, voice simulations were extended in 4.1.10, and TranscriptionAccuracyMetric, AudioGlitchMetric and audio input for judge models arrived in the October 2026 release. This page was verified against deepeval==4.3.1 on Python 3.13. Earlier versions (this guide's base was 3.8.4) do not have these classes.

How It Works¶

flowchart LR
  G["ConversationalGolden<br/>scenario + expected_outcome"] --> S["ConversationSimulator<br/>(voice_config=...)"]
  S -->|"TTS: user speech"| C["Voice connector<br/>ElevenLabs / LiveKit / Vapi /<br/>Pipecat / WebSocket / Callback"]
  C -->|"agent audio"| S
  S --> T["ConversationalTestCase<br/>turns with Audio, latency_ms,<br/>start_time"]
  T --> M["Voice metrics<br/>+ conversation metrics"]

1. You write a ConversationalGolden (scenario and expected outcome), as for a text chatbot (Part 5).
2. ConversationSimulator plays the caller: an LLM writes the user lines, a text-to-speech (TTS) model speaks them, and a connector delivers the audio to your agent and brings the reply back.
3. Every turn is stored with its audio and timing in a ConversationalTestCase.
4. Voice metrics read the audio and the timeline; ordinary conversation metrics (GoalAccuracyMetric, TurnRelevancyMetric, ToolUseMetric) read the transcript.

Data Model¶

Audio¶

from deepeval.test_case import Audio

recording = Audio(url="./agent-reply.wav")                       # file or URL
recording = Audio.from_bytes(                                    # raw bytes
    wav_bytes, "audio/wav", sampleRate=24000, duration=3.2, start_time=2.1,
)

Turn fields used by voice metrics¶

Running a Simulated Call¶

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import AudioIntegrityMetric, VoiceNaturalnessMetric, VoiceReliabilityMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig

golden = ConversationalGolden(
    scenario="The user wants to change a restaurant reservation.",
    expected_outcome="The agent confirms the new reservation date.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(connector=ElevenLabsConnector(agent_id="your-agent-id")),
)
test_cases = simulator.simulate([golden])          # runs the call, records the audio

evaluate(
    test_cases=test_cases,
    metrics=[
        VoiceReliabilityMetric(threshold=0.9),
        VoiceNaturalnessMetric(threshold=0.7),
        AudioIntegrityMetric(threshold=0.8),
    ],
)

This block needs a live agent and an API key (the simulator's user-LLM and TTS default to OpenAI), so it was not run here. The metric part below was run end to end on synthetic audio.

Connectors¶

CallbackVoiceConnector is the easiest way to start and to test your own code path:

from deepeval.test_case import Audio
from deepeval.voice import CallbackVoiceConnector, VoiceConfig


def my_agent(user_audio: Audio) -> Audio:
    reply_text = run_my_pipeline(user_audio)       # your STT → LLM → TTS
    return synthesize(reply_text)                   # returns Audio


config = VoiceConfig(connector=CallbackVoiceConnector(my_agent, sample_rate=24000, turn_detection="balanced"))

VoiceConfig also takes tts_model / stt_model (strings or DeepEvalBaseTTS / DeepEvalBaseSTT subclasses), output_dir for the saved audio, record_call and interruption_settings.

Requirements and limits¶

• The simulator's default TTS and STT are OpenAI models (the STT default is gpt-4o-transcribe). Environment variables DEEPEVAL_TTS_MODEL and DEEPEVAL_STT_MODEL select other models; other providers need DeepEvalBaseTTS / DeepEvalBaseSTT implementations.
• A connector holds one live call, so voice conversations run one after another, not in parallel.
• Interruptions (barge-in) need a full-duplex connector such as LiveKit and must be enabled on the persona; the default is half-duplex with no interruptions.
• End-of-turn detection is heuristic. Latency is comparable only between runs on the same connector type.
• Saved call audio goes to a configurable output folder and is excluded from version control by default.
• STT mistakes count as agent mistakes: transcription quality caps what you can measure.

The Voice Metrics¶

All return a score from 0 to 1, higher is better, and all expose score_breakdown after measure(). Parameters common to all of them: threshold (default 0.5, None = score only), include_reason=True, strict_mode=False, verbose_mode=False, flaky=False.

Seven of them need no LLM and cost no tokens, so they are safe to run on every call in CI. AudioGlitchMetric needs a judge model object only to write the reason (see Gotchas), and TranscriptionAccuracyMetric is judge-based.

Scoring details¶

Running them offline on audio you already have¶

You do not need a live agent to evaluate recordings. Build a ConversationalTestCase yourself (from call logs, QA recordings or the output of a simulation):

from deepeval.test_case import Audio, ConversationalTestCase, Turn
from deepeval.metrics import (
    AgentResponsivenessMetric,
    AudioIntegrityMetric,
    TurnTakingNaturalnessMetric,
    VoiceNaturalnessMetric,
    VoiceReliabilityMetric,
)

tc = ConversationalTestCase(
    scenario="User asks for an order status.",
    turns=[
        Turn(role="user", content="Where is my order?",
             audio=Audio.from_bytes(user_wav, "audio/wav", sampleRate=16000, duration=1.5, start_time=0.0)),
        Turn(role="assistant", content="Your order shipped yesterday and arrives on Friday afternoon.",
             audio=Audio.from_bytes(agent_wav, "audio/wav", sampleRate=16000, duration=4.3, start_time=2.1),
             latency_ms=600),
    ],
)

for metric in (VoiceNaturalnessMetric(threshold=0.7), AudioIntegrityMetric(threshold=0.7),
               AgentResponsivenessMetric(threshold=0.7), TurnTakingNaturalnessMetric(threshold=0.7),
               VoiceReliabilityMetric(threshold=0.7)):
    metric.measure(tc)
    print(type(metric).__name__, round(metric.score, 2), metric.success)

To check the metrics themselves, the examples below were run on synthetic audio: speech-like bursts of a 150-250 Hz tone with short pauses (about 4 s), written with the wave module. A "good" agent reply is clean; a "clipped" reply has the same signal driven 6× into the ceiling. Results with threshold=0.7:

The breakdown explains the clipped score:

metric = VoiceNaturalnessMetric(threshold=0.7); metric.measure(clipped_call)
print(metric.score_breakdown["turns"][0])
# {'turn': 1, 'score': 0.65, 'speaking_rate_wpm': 177.6, 'silence_fraction': 0.29,
#  'pitch_variation_hz': 29.7, 'clipping_fraction': 0.157, 'dropout_events': 0, 'loop_events': 0}

metric = AudioIntegrityMetric(threshold=0.7); metric.measure(clipped_call)
print(metric.score_breakdown)
# {'critical_failure': False, 'events': [{'type': 'clipping', 'turn': 1, 'fraction': 0.157, 'severity': 0.35, 'critical': False}]}

Note VoiceReliabilityMetric still passes 0.7 at 0.82 because clipping is non-critical and responsiveness is perfect. Choose thresholds per metric, and use VoiceReliabilityMetric as a gate for critical failures rather than as a quality score.

Metrics catch different defects¶

A second synthetic call: the agent's reply has 1.2 s of digital silence (an exact-zero stream) in the middle.

AudioGlitchMetric caught it, while the other two did not. Run several voice metrics together: each one looks for different failures.

Use them in pytest¶

import pytest
from deepeval import assert_test


@pytest.mark.parametrize("call", recorded_calls)
def test_voice_quality(call):
    assert_test(test_case=call, metrics=[VoiceReliabilityMetric(threshold=0.9), AudioGlitchMetric(threshold=0.8)])

A failing call raises AssertionError: Metrics: Audio Integrity (score: 0.65, threshold: 0.9, ...) failed. Run with deepeval test run test_voice.py.

Gotchas¶

Evaluation Strategy¶

1. Smoke gate (every commit): VoiceReliabilityMetric + AudioGlitchMetric over a handful of simulated calls: catches hangups, missing audio, dead streams.
2. Quality (nightly): naturalness, intelligibility, consistency and turn-taking over 20+ scenarios; track the trend per connector.
3. Content: the usual conversation metrics (GoalAccuracyMetric, ToolUseMetric, TurnRelevancyMetric) on the same test cases, see Part 5 and Part 20.
4. Hearing: TranscriptionAccuracyMetric when the platform exposes the agent's own transcript: it separates "the agent misheard" from "the agent answered badly".
5. Calibrate on real calls: play 10 recordings, mark good and bad ones, and set thresholds where the metric agrees with the listener.

Sources¶

• DeepEval — Voice evaluation
• Voice Naturalness, Voice Reliability, Voice Consistency
• Audio Integrity, Audio Glitch, Transcription Accuracy
• Turn-Taking Naturalness, Speech Intelligibility, Agent Responsiveness
• DeepEval releases on GitHub, 2026 changelog

See also¶

• DeepEval Guide — Part 5: Chatbot Metrics
• DeepEval Guide — Part 8: Multimodal (Image) Metrics
• DeepEval Guide — Part 21: What's New in DeepEval 4.x
• DeepEval Guide — Part 22: Classifiers & Expectations
• DeepEval — LLM Testing Guide