• October 25, 2026
• 8 min read

promptfoo — Setup & Config¶

Installing the CLI, the shape of promptfooconfig.yaml, how prompts are written and templated, where API keys come from, what the cache does and which flags matter when running promptfoo eval.

Create a Project¶

mkdir ticket-evals && cd ticket-evals
npx promptfoo@latest init --no-interactive      # writes promptfooconfig.yaml + README.md
npx promptfoo@latest init --example getting-started   # or download an official example
npx promptfoo@latest init                       # interactive wizard

promptfoo eval reads promptfooconfig.yaml from the current directory. .yaml, .yml, .json and .js configs are supported; pass another file with -c path/to/config.yaml. Several -c files (or a glob) are merged into one eval.

Config Anatomy¶

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Support bot regression suite
tags:
  suite: smoke                     # eval tags, shown in the UI and exports

prompts:                           # what is sent
  - file://prompts/support.txt
  - file://prompts/support_chat.json

providers:                         # who answers (alias: targets)
  - id: openai:chat:gpt-6-luna
    label: luna
    config:
      temperature: 0

defaultTest:                       # merged into every test
  vars:
    company: Acme
  options:
    provider: openai:gpt-6-sol     # grader for model-graded assertions
  assert:
    - type: latency
      threshold: 5000

tests: file://tests/cases.csv      # or an inline list

evaluateOptions:
  maxConcurrency: 4
  repeat: 1
  cache: true

sharing: false                     # never upload automatically
outputPath: out/results.json       # same as -o

Providers, tests and defaults can live in separate files:

providers:
  - file://providers/gateway.yaml
tests:
  - file://tests/*.yaml
  - vars: { question: Inline test next to file-based ones }
defaultTest: file://tests/defaults.yaml

Prompts¶

[
  {"role": "system", "content": "You are a support bot for {{company}}. Be brief."},
  {"role": "user", "content": "{{question}}"}
]

// prompts/build.js — return a string, a messages array or {prompt, config}
module.exports = async function ({ vars, provider }) {
  return [
    { role: 'system', content: `Support bot for ${vars.company}.` },
    { role: 'user', content: vars.question },
  ];
};

# prompts/build.py
import json

def build_prompt(context):
    vars = context["vars"]
    return json.dumps([
        {"role": "system", "content": f"Support bot for {vars['company']}."},
        {"role": "user", "content": vars["question"]},
    ])

Templating rules worth knowing:

• Values in vars can be strings, numbers, objects or arrays. Arrays are expanded into the cross product of tests unless options.disableVarExpansion: true.
• In a prompt that is valid JSON, variables are escaped automatically. In a prompt that is not valid JSON (loops, {% if %}), use {{ value | dump }}.
• {{ env.NAME }} reads environment variables when the config is loaded.
• nunjucksFilters: {shout: ./filters/shout.js} registers {{ text | shout }}.
• With no prompts, {{prompt}} is used as a passthrough.
• Prompt files can be pulled from langfuse://my-prompt@production, portkey://... and helicone://....

Environment Variables and API Keys¶

Without --env-file, promptfoo loads .env from the current directory; values already set in the shell win. --env-file .env --env-file .env.ci (or comma-separated) loads several files, later ones override. DOTENV_PATH names another default file and DOTENV_OVERRIDE=true lets the file beat the shell. Per-provider apiKey, apiKeyEnvar and apiBaseUrl live under providers[].config.

Caching¶

Successful provider responses are cached on disk (~/.promptfoo/cache, 14 days). Errors and empty responses are not cached. The key includes provider, rendered prompt, provider config and context variables, so changing any of them is a miss.

Use --no-cache for latency and cost assertions, for fresh model behaviour, and with --repeat (each repeat index has its own cache namespace).

Running Evals¶

promptfoo eval                                    # config from the current directory
promptfoo eval -c evals/triage.yaml -o results.json -o report.html
promptfoo eval --var company=Globex --repeat 3 --no-cache
promptfoo eval -p 'Q: {{question}}' -r openai:gpt-6-luna -t questions.csv    # no config file
promptfoo eval --assertions asserts.yaml --model-outputs outputs.json        # score saved outputs

In eval, -v means --vars (CSV path); for debug output use --verbose or LOG_LEVEL=debug.

Exit codes: 0 all tests passed, 100 at least one failed or the pass rate is below PROMPTFOO_PASS_RATE_THRESHOLD (percent, default 100), 1 any other error. PROMPTFOO_FAILED_TEST_EXIT_CODE replaces 100.

Web UI, Data and Telemetry¶

• promptfoo view serves the results on port 15500 (-p changes it, -y skips the open-browser prompt). The UI filters failures, shows full prompts, compares evals and exports CSV / JSON / YAML.
• Every eval is stored in a local SQLite database, ~/.promptfoo/promptfoo.db, next to cache/ and logs/. PROMPTFOO_CONFIG_DIR relocates the directory (then promptfoo view <dir>); PROMPTFOO_LOG_DIR relocates logs. promptfoo list evals, show, export eval <id>, import and delete manage stored runs.
• Telemetry (command names, assertion types, package version, CI flag; no prompts, outputs or keys) is on by default. Disable with PROMPTFOO_DISABLE_TELEMETRY=1; disable the npm update check with PROMPTFOO_DISABLE_UPDATE=1.
• Results go to a shared URL only through --share, sharing: true / sharing: {apiBaseUrl, appBaseUrl} or promptfoo share. --no-share, sharing: false and PROMPTFOO_DISABLE_SHARING=true block uploads; shared snapshots include prompts, variables and outputs.

# CI and Docker defaults worth setting
export PROMPTFOO_DISABLE_TELEMETRY=1 PROMPTFOO_DISABLE_UPDATE=1
export PROMPTFOO_CACHE_PATH="$HOME/.cache/promptfoo"
export PROMPTFOO_PYTHON=python3

See also¶

• promptfoo — LLM Evals & Red Teaming from YAML
• promptfoo — Providers
• promptfoo — Test Cases & Assertions
• Langfuse — Prompt Management
• Docker & Docker Compose — Overview
• Eval Harness — Tools, Testing & CI