• October 19, 2026
• 10 min read

DeepEval Guide — Part 21: What's New in DeepEval 4.x¶

The rest of this guide was written against deepeval==3.8.4 (February 2026). By October 2026 the current release is 4.3.1. This page lists what changed, what breaks when you upgrade, and the new evals (voice, classifiers, expectations, deterministic metrics) with pointers to the pages that cover them.

How this page was built: the official 2026 changelog, the GitHub releases and the DeepEval 4.0 post, plus a check of every claim that could be checked against the installed deepeval==4.3.1 (imports, signatures, source, and offline runs). Items I could only read about and not confirm in code are marked.

Timeline¶

Breaking and Behavior Changes (Migration)¶

The score inversion in detail¶

In 4.3.1 ToxicityMetric, BiasMetric, MisuseMetric, NonAdviceMetric and PIILeakageMetric compute the score as the share of verdicts that are good (Verdict.NO for "yes is bad" questions), and HallucinationMetric as the share of context items the answer agrees with. The base is_successful() is score >= threshold. Consequences:

# deepeval 3.x: low score = good, pass when score <= threshold
ToxicityMetric(threshold=0.2)        # allows up to 20% toxic opinions

# deepeval 4.2+: high score = good, pass when score >= threshold
ToxicityMetric(threshold=0.8)        # requires at least 80% non-toxic opinions

Dashboards, CI gates and your own assert metric.score < 0.3 checks all need the same flip. Search your code for Hallucination, Toxicity, Bias and Misuse together with score <. Pages Part 2 and Part 3 of this guide show both behaviors.

Pin and upgrade deliberately¶

uv add "deepeval==4.3.1"          # exact pin; the project ships many releases a month
uv run deepeval --version         # 4.3.1

One release (4.0.8) was yanked from PyPI because of a broken login flow; use 4.0.9 or newer.

New Evals and Metrics¶

Voice evaluation (4.2.0+)¶

Simulated phone calls with recorded audio, and nine metrics that score the call from the audio signal. Seven need no LLM at all. See Part 8b: Voice Evaluation.

Classifiers and Expectations (4.2.4+ / 4.3.1)¶

Closed-label judges (RefusalClassifier, PromptInjectionClassifier, ...) and must / must_not sentences on a test case. See Part 22: Classifiers & Expectations.

ToolPermissionMetric (4.0.8) — deterministic¶

An allowlist/denylist check on tools_called. No LLM, no API key. Run offline on 4.3.1:

from deepeval.metrics import ToolPermissionMetric
from deepeval.test_case import LLMTestCase, ToolCall

tc = LLMTestCase(
    input="clean up", actual_output="done",
    tools_called=[ToolCall(name="read_file", input_parameters={"path": "a"}),
                  ToolCall(name="delete_file", input_parameters={"path": "a"})],
)

ToolPermissionMetric(allowed_tools=["read_file", "list_dir"])      # allowlist
ToolPermissionMetric(denied_tools=["delete_file"])                 # denylist

The default threshold is 1.0: one unauthorized call fails the case. Use it as a guardrail test for coding agents and tool-using agents.

AgentLoopDetectionMetric (4.0.8) — deterministic, trace-based¶

Detects an agent stuck in a loop. It reads the trace, not tools_called, so the app must be instrumented with @observe:

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AgentLoopDetectionMetric
from deepeval.tracing import observe, update_current_trace


@observe(type="tool")
def search(q: str) -> str:
    return "no results"


@observe()
def agent(q: str) -> str:
    for _ in range(5):
        search(q)                       # same tool, same arguments
    update_current_trace(input=q, output="gave up")
    return "gave up"


dataset = EvaluationDataset(goldens=[Golden(input="find x")])
for golden in dataset.evals_iterator(metrics=[AgentLoopDetectionMetric(threshold=0.9)]):
    agent(golden.input)

Three checks, weighted: tool-call repetition 40% (same name and arguments at least repetition_threshold=3 times), reasoning stagnation 35% (consecutive LLM outputs with similarity ≥ similarity_threshold=0.85), call-graph cycles 25% (a span repeated on one root-to-leaf path). Disabled checks (check_tool_repetition=False, ...) do not count against the score.

Run on 4.3.1: five identical search calls scored 0.80 and the report said "Tool 'search' called 5 times with identical arguments"; a trace with one call scored 1.00. With threshold=0.7 the looping run passed, with 0.9 it failed, so set the threshold deliberately. Without trace data the metric returns 0 with No trace data found. This metric requires trace data from @observe. The metric instance you pass to evals_iterator() is copied, so read scores from the report rather than from your object.

CitationFaithfulnessMetric (listed in the 4.0.8 changelog)¶

The changelog announces it for citation misattribution. I could not find it in deepeval.metrics of 4.3.1 (the 58 exported *Metric classes include AgentLoopDetectionMetric and ToolPermissionMetric but no citation metric), so check the docs and your installed version before depending on it.

Other metric changes¶

Eval Modes and Jev (4.2.4+)¶

LLM-as-a-judge metrics can now be configured by who decides:

Select it per metric (HallucinationMetric(eval_mode="hybrid")), globally with deepeval set-eval-mode, or with the DEEPEVAL_EVAL_MODE environment variable. Precedence: the metric argument, then the setting, then llm. Classifiers and Expectations accept only llm and system_one. Jev needs the TypeSafe SDK and an API key; a missing key or SDK fails when the metric is constructed. Nothing turns it on implicitly: existing suites behave as before. The release notes label Jev opt-in and experimental, so treat scores from hybrid and system_one as a separate baseline from llm.

CLI¶

deepeval --help on 4.3.1 lists these commands (selection):

deepeval inspect plus local trace storage is the main 4.0 idea: a coding agent (Claude Code, Codex, Cursor) can run deepeval test run, read the saved traces and iterate. DeepEval publishes agent skills for Cursor and Claude in its repository.

evaluate() and DisplayConfig gained report options: HTML and Markdown exports, truncation of passing cases (truncate_passing_cases=True), inspect_after_run=True, results_folder. assert_test() now takes classifiers=; evaluate() takes classifiers=, mcp_servers= and official= (baseline runs, needs CONFIDENT_API_KEY).

Tracing and Integrations¶

One-line tracing and evaluation for several frameworks, plus OpenTelemetry export:

New judge-model providers: OpenRouterModel, Gemini 3 series, Vertex AI with user credentials, Amazon Bedrock fixes, claude-opus-4-8 and gpt-5.4-mini.

Datasets, Goldens and Simulation¶

Upgrade Checklist¶

1. Pin deepeval==4.3.1 in a branch and run your suite once unchanged; note every flipped result.
2. Flip thresholds for Bias, Hallucination, Misuse and Toxicity (and any code comparing their scores).
3. Replace observed_callback, API_KEY and additional_metadata usage; rename controller= to stopping_controller=.
4. Re-baseline TurnRelevancy, KnowledgeRetention, Summarization and GEval scores: wording and weighting changed.
5. Add deterministic guardrails that cost nothing: ToolPermissionMetric, AgentLoopDetectionMetric, JsonCorrectnessMetric, PatternMatchMetric.
6. Add classifiers for exact outcomes (refusal, prompt injection, data leakage) and Expectations for plain-language acceptance criteria.
7. If you ship a voice agent, add the voice metrics in Part 8b to the smoke suite.
8. Set DEEPEVAL_TELEMETRY_OPT_OUT=YES in CI if you do not want telemetry, and decide on deepeval login for result history.

Sources¶

• DeepEval 2026 changelog
• GitHub releases
• Introducing DeepEval 4.0
• Introduction to LLM Evals
• Agent Loop Detection
• deepeval on PyPI

See also¶

• DeepEval Guide — Part 8b: Voice Evaluation
• DeepEval Guide — Part 22: Classifiers & Expectations
• DeepEval Guide — Part 18: Evaluation Configs, Flags & Reference
• DeepEval Guide — Part 4: AI Agent Metrics
• DeepEval — LLM Testing Guide