• October 22, 2026
• 5 min read

LangSmith — LLM Tracing, Prompts & Evals¶

LangChain's platform for tracing, evaluating and versioning LLM applications and agents. It works with or without LangChain: the Python SDK (langsmith, MIT) instruments any code, and the platform runs as a managed cloud, in your own cloud, or on your Kubernetes cluster.

What LangSmith Gives a QA Engineer¶

Installation¶

uv add langsmith                  # SDK: tracing, datasets, evaluate(), prompts
uv add "langsmith[pytest]"        # pytest plugin extras: rich output, HTTP caching
uv add "langsmith[otel]"          # OpenTelemetry integration
uv add openevals                  # prebuilt evaluators (LLM-as-judge, exact match, ...)

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_pt_..."
export LANGSMITH_PROJECT="support-bot-ci"                     # default: "default"
# export LANGSMITH_ENDPOINT="https://eu.api.smith.langchain.com"   # non-US region or self-hosted

Data Model¶

flowchart TD
  ORG["Organization"] --> WS["Workspace<br/>members, API keys, datasets, prompts"]
  WS --> TP["Tracing project<br/>(or Agent → Environment)"]
  TP --> TH["Thread<br/>thread_id / session_id metadata"]
  TH --> T1["Trace<br/>one request / test case"]
  TP --> T1
  T1 --> R1["Run: chain triage"]
  R1 --> R2["Run: retriever search-kb"]
  R1 --> R3["Run: llm classify<br/>tokens, cost"]
  FB["Feedback<br/>key + score / value"] -.-> R1
  FB -.-> R3
  WS --> DS["Dataset → Examples<br/>inputs, reference outputs"]
  DS --> EXP["Experiment<br/>app version run over a dataset"]
  EXP --> T1
  WS --> PR["Prompt → commits → tags"]

Section Map¶

Minimal Trace¶

from langsmith import traceable, Client
from langsmith.wrappers import wrap_openai
import openai

oai = wrap_openai(openai.OpenAI())          # every call becomes an llm run with token usage


@traceable(name="triage", tags=["smoke"], metadata={"thread_id": "t-1"})
def triage(ticket: str) -> str:
    resp = oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Classify priority P1-P4: {ticket}"}],
    )
    return resp.choices[0].message.content


print(triage("Checkout returns 500 for all EU users"))
Client().flush()                            # short-lived script: send buffered runs before exit

Quick Commands¶

LangSmith vs Langfuse¶

Full decision table including Phoenix and MLflow: 05 Testing, CI & Production.

Quick Rules¶

1. Set LANGSMITH_TRACING=true explicitly — decorators and wrappers do nothing without it; it is also the off switch for tests.
2. Match LANGSMITH_ENDPOINT to the region where the account was created; a US endpoint rejects an EU key.
3. Separate test traffic: a dedicated project (or the local / development agent environment) for CI, short retention.
4. Put thread_id metadata on every run of a conversation, not only the root.
5. Always flush (Client().flush()) in scripts, jobs and serverless handlers.
6. Pull prompts by tag (name:production), never hard-code a commit hash in production code.
7. Turn bad production runs into dataset examples — the dataset is the regression suite.
8. Gate CI on aggregate scores (summary evaluators, thresholds), not on exact string matches.
9. Mask PII in the SDK (anonymizer, hide_inputs, process_inputs) — runs are sent raw by default.
10. Use service or tracing keys for apps and CI, not personal access tokens.

See also¶

• Digital Garden: Knowledge Base
• Tools — Practical Reference Guides
• Langfuse — LLM Tracing, Prompts & Evals
• Arize Phoenix
• MLflow — Experiment Tracking, LLM Tracing & Evaluation
• Jaeger — Distributed Tracing for OpenTelemetry
• LangChain — LLM Application Framework
• LangGraph — Stateful Agent Orchestration
• OpenTelemetry — Python Observability
• DeepEval — LLM Testing Guide