• September 29, 2026
• 6 min read

MLflow — Experiment Tracking, LLM Tracing & Evaluation¶

Open-source (Apache 2.0, Linux Foundation) platform for the ML and GenAI lifecycle: experiment tracking, a model registry, and — since MLflow 3 — tracing, evaluation and a prompt registry for LLM apps and agents. One server with a UI and REST API, backed by a SQL database and an artifact store.

What MLflow Gives a QA Engineer¶

1. A history of every test and eval run — params, metrics, tags and report files per CI run, searchable and comparable in the UI.
2. LLM traces — autologging for OpenAI, Anthropic, LangChain / LangGraph, LiteLLM and others; @mlflow.trace for your own code.
3. Evaluation — mlflow.genai.evaluate() with built-in LLM judges, custom scorers, DeepEval metrics as scorers, versioned datasets.
4. Release gates — prompt and model versions with aliases (@candidate, @production), promoted only when the eval run passes.

Where MLflow Fits¶

flowchart LR
  subgraph Client["Test suite / LLM app"]
    SDK["mlflow SDK<br/>runs, metrics, artifacts"]
    TR["mlflow tracing<br/>autolog, @mlflow.trace"]
    EV["mlflow.genai.evaluate<br/>scorers, LLM judges"]
    OT["Any OTel SDK"]
  end
  SDK -- "REST" --> S["MLflow server :5000<br/>UI + REST API"]
  TR -- "REST" --> S
  EV -- "REST" --> S
  OT -- "OTLP HTTP /v1/traces" --> S
  S --> DB[("Backend store<br/>SQLite / PostgreSQL / MySQL")]
  S --> AR[("Artifact store<br/>local dir / S3 / GCS / Azure")]
  TR -. "optional OTLP export" .-> J["Jaeger / OTel Collector"]

Section Map¶

Installation¶

uv add mlflow                  # full package: SDK, server, UI, evaluation
uv add mlflow-tracing          # lightweight tracing-only SDK for production services

mlflow-tracing contains only the tracing SDK with a minimal set of dependencies — enough for an app that sends traces to a remote MLflow server. The server, UI and evaluation need the full mlflow package.

Minimal Setup¶

uv run mlflow server --port 5000        # SQLite ./mlflow.db + ./mlartifacts, UI on http://localhost:5000

# uv add mlflow openai
import mlflow
from openai import OpenAI

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("support-bot")
mlflow.openai.autolog()                  # every OpenAI call becomes a trace

with mlflow.start_run(run_name="smoke"):
    mlflow.log_param("model", "gpt-4.1-mini")
    reply = OpenAI().chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": "What is the capital of France?"}],
    )
    mlflow.log_metric("answer_chars", len(reply.choices[0].message.content))
# UI -> Experiments -> support-bot: the run on the Runs tab, the LLM call on the Traces tab

Quick Commands¶

Key Environment Variables¶

MLflow vs Phoenix, Langfuse and Jaeger¶

Rule of thumb: pick MLflow when you want test and eval runs as first-class, comparable records and a registry to gate releases; pick Phoenix or Langfuse when LLM observability is the only need; keep Jaeger for non-LLM service traces.

Quick Rules¶

1. Always set MLFLOW_TRACKING_URI in CI — without it every job writes to a throwaway local mlflow.db.
2. One experiment per suite, one run per CI job; the run name carries the pipeline id.
3. Tag every run with git_sha, git_branch, ci_pipeline_id — regressions are found by these tags.
4. Params for inputs, metrics for results — params are immutable strings; metrics are numbers with history.
5. Autolog once per framework (mlflow.openai.autolog()) and add @mlflow.trace only for your own code — not on top of autologged calls.
6. Prefer code scorers, then built-in judges, then custom judges — and pin the judge model.
7. Use aliases, not version numbers, for prompts and models that tests and apps load.
8. Pin the server version and run mlflow db upgrade on purpose when upgrading.

See also¶

• Digital Garden: Knowledge Base
• Tools — Practical Reference Guides
• DeepEval — LLM Testing Guide
• Arize Phoenix — LLM Tracing & Evaluation
• Langfuse — LLM Tracing, Prompts & Evals
• Jaeger — Distributed Tracing for OpenTelemetry
• OpenTelemetry — Python Observability
• Eval Harness — Tools, Testing & CI