• September 29, 2026
• 5 min read

MLflow — Experiment Tracking¶

Experiment tracking is the original core of MLflow: every execution is a run with its inputs (params), results (metrics), labels (tags) and files (artifacts). For a QA engineer a run is a test or eval execution — one CI job, one benchmark, one prompt variant — and the experiment is the history of all of them.

Data Model¶

flowchart TD
  E["Experiment<br/>llm-regression"] --> R1["Run ci-1234<br/>status FINISHED"]
  E --> R2["Run ci-1235"]
  R1 --> P["Params<br/>model=gpt-4.1-mini<br/>prompt_version=v3"]
  R1 --> M["Metrics<br/>pass_rate=0.97<br/>faithfulness_mean=0.91"]
  R1 --> T["Tags<br/>git_sha, git_branch,<br/>ci_pipeline_id"]
  R1 --> A["Artifacts<br/>report.xml, eval_results.json"]

Logging a Run¶

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")   # or MLFLOW_TRACKING_URI
mlflow.set_experiment("checkout-api-tests")        # created on first use

with mlflow.start_run(run_name="ci-1234") as run:
    mlflow.set_tags({"git_sha": "a1b2c3d", "ci_pipeline_id": "1234", "suite": "smoke"})
    mlflow.log_params({"base_url": "https://staging.example.com", "workers": 4})
    mlflow.log_metrics({"tests_total": 120, "tests_failed": 3, "pass_rate": 0.975})

    for step, latency in enumerate([120, 135, 128]):          # a series, plotted as a chart
        mlflow.log_metric("p95_latency_ms", latency, step=step)

    mlflow.log_dict({"failed": ["test_refund", "test_cancel"]}, "failures.json")
    mlflow.log_text("staging, feature flags: new-checkout", "notes/env.txt")
    mlflow.log_artifact("report.xml", artifact_path="reports")  # an existing local file
    print(run.info.run_id, run.info.experiment_id)

• The with block ends the run as FINISHED, or as FAILED if an exception escapes it.
• The run is active only in this process and thread; mlflow.log_* outside a run starts a new one implicitly — always use start_run() explicitly.
• log_params values are converted to strings: 4 becomes "4", so filter as params.workers = '4'.

Artifact Helpers¶

System Tags¶

MLflow sets some tags itself: mlflow.runName, mlflow.user, mlflow.source.name, mlflow.source.type and — when the entry-point script lives in a git repository — mlflow.source.git.commit and mlflow.source.git.branch.

Autologging¶

mlflow.autolog() patches supported libraries so params, metrics and models are logged without explicit calls.

import mlflow
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

mlflow.set_experiment("iris")
mlflow.autolog()                       # or only one library: mlflow.sklearn.autolog()

X, y = load_iris(return_X_y=True)
with mlflow.start_run():
    LogisticRegression(max_iter=200).fit(X, y)
# Params: C, max_iter, ...; metrics: training_accuracy_score, training_f1_score, ...; the model as an artifact

Searching Runs¶

import mlflow

runs = mlflow.search_runs(
    experiment_names=["checkout-api-tests"],
    filter_string="tags.suite = 'smoke' and metrics.pass_rate < 0.99",
    order_by=["attributes.start_time DESC"],
    max_results=20,
)                                              # pandas.DataFrame: run_id, metrics.*, params.*, tags.*
print(runs[["run_id", "metrics.pass_rate", "tags.git_sha"]])

latest = mlflow.search_runs(
    experiment_names=["checkout-api-tests"],
    filter_string="attributes.status = 'FINISHED'",
    order_by=["attributes.start_time DESC"],
    max_results=1,
    output_format="list",                      # list[Run] instead of a DataFrame
)[0]
print(latest.data.metrics["pass_rate"], latest.data.tags["git_sha"])

Filter syntax essentials:

For metric history (all steps) use MlflowClient().get_metric_history(run_id, "p95_latency_ms").

Comparing Runs in the UI¶

1. Experiments → your experiment → Runs — the table shows params, metrics and tags as columns; the search box takes the same filter syntax.
2. Select two or more runs → Compare — side-by-side params and metrics, parallel-coordinates and scatter plots, metric charts over steps.
3. Chart view — one chart per metric across runs, good for "pass rate over the last 30 builds".
4. Artifacts tab of a run — download reports, preview JSON, text, images and tables.

Naming Conventions for Test Runs¶

Tracking Checklist¶

• One experiment per suite; experiment name comes from MLFLOW_EXPERIMENT_NAME in CI
• Every run has git_sha, git_branch and ci_pipeline_id tags
• Everything that changes the result is a param (model, prompt version, dataset version, judge)
• Metrics are aggregates with stable names; per-case detail goes to a table artifact
• Reports (JUnit XML, Allure, HTML) are attached as artifacts
• Runs are always started explicitly with start_run() inside a with block

See also¶

• MLflow — Experiment Tracking, LLM Tracing & Evaluation
• MLflow — Setup & Architecture
• MLflow — GenAI Tracing
• MLflow — Testing, CI & Model Registry
• Pytest — Python Testing Framework