LangSmith vs Braintrust vs Helicone vs Arize Phoenix: Four Loops the Eval/Observability Stack Was Built to Close

At a glance

Four products, four answers to the same question — at which point in the lifecycle does the team actually look at this data and act on it. The table below sets the basics; the matrix that follows shows where each one leans hardest across the axes that actually differ.

Snapshot: 2026-06-01. These platforms ship frequently; verify against current docs.

LangSmith — deep dive

Data model — runs, traces, and prompt-versioned datasets

A LangSmith run is a tree of nested child runs — one node per LLM call, tool call, retriever invocation, or chain step — with inputs, outputs, latency, token usage, cost, and free-form metadata on each node. The same schema describes a unit test, a CI evaluation, and a production request, which is what lets a flagged production trace get promoted into a dataset row without remodelling anything. Datasets are versioned and forkable, so "the golden 200 examples we run prompt changes against" is a first-class object you can pin to a commit.

The loop it closes — prompt hub → dataset → online eval → prompt hub

LangSmith's centre of gravity is the prompt-iteration loop for teams building on LangChain or LangGraph. You write a prompt in the Prompt Hub, run it against a dataset in the Playground, eyeball per-example diffs, score with built-in or custom evaluators, then promote the winning version into the app. Online evaluators close the back half of the loop: sampled production traces get scored asynchronously, and regressions surface as new dataset rows. The same shape supports a managed LangGraph runtime under the LangSmith Deployment umbrella, so the dev loop and the prod loop share a control plane. The pull is strongest if your app is already framework-shaped — graphs, chains, agents wired through the LangChain stack get instrumented for free.

Integration shape — SDK callbacks plus OTel support

The primary integration is the LangChain tracer callback: if you already use LangChain or LangGraph, instrumentation is one env var. Outside that, the @traceable decorator wraps arbitrary Python or JS functions into LangSmith spans, and the platform also ingests OpenTelemetry data via OTLP, so a team standardising on OTel is not locked out. The trade-off is real though: outside the LangChain world you get traces, but you give up some of the framework-aware UI affordances — agent-step grouping, prompt-version linkage — that are the reason to pick LangSmith in the first place.

Braintrust — deep dive

Data model — Eval-as-code and immutable experiments

The unit you author in Braintrust is an Eval() — a function that pairs a dataset, a task (your prompt or chain), and a list of scorers, all written in TypeScript or Python and checked into your repo. A run of that Eval() produces an experiment: an immutable build artifact stitching every example, every model output, every scorer score, and aggregate metrics into one object you can permalink, compare, and store next to the commit that produced it. Datasets and scorers are first-class; both are versioned. The autoevals library ships a battery of LLM-judges, classification, and similarity scorers that you import like any other dependency.

The loop it closes — every pull request is an eval run

This is the CI loop. braintrust eval ./eval.ts runs in GitHub Actions on every PR, the resulting experiment is diffed against the baseline experiment on main, and the per-example regressions show up as a PR comment with side-by-side outputs. A drop in accuracy on the golden set, a spike in cost-per-example, a latency regression — any of them can fail the check and block the merge. That is a different muscle than browsing a trace dashboard after the fact: it forces a per-PR answer to "did this change help or hurt," exactly the muscle that Evals 101 calls the bare minimum for non-toy LLM work. The principle scales down to RAG too — see Evaluating RAG for what an eval set for retrieval/grounding/answer-quality actually looks like.

Integration shape — wrappers and online logging

Instrumentation is opt-in code rather than auto-magic: wrapOpenAI() wraps a client, @traced wraps a function, and spans nest naturally. There is online logging too — production calls stream into Braintrust and can seed new dataset rows, so a thumbs-down in the UI becomes the next regression test on the next PR. But the centre of gravity is local-first: evals run on your laptop and in CI before they run in prod, which is the inverse of a gateway-first product.

Helicone — deep dive

Data model — proxied requests as the unit of observation

The Helicone unit is an HTTP request to the model provider, captured at the gateway. Each row carries the full prompt, the full response, the streaming chunks, latency, token counts, computed cost, model, provider, user/session/request tags, and any custom headers you set. Multi-step agents and chains group via the session ID — a header you set per logical "conversation" or "agent run" — so a multi-tool agent loop shows up as a session tree instead of forty unrelated requests. There is no graph-of-nodes abstraction baked in; the gateway sees what the gateway sees.

The loop it closes — production observability without an SDK migration

Helicone's pitch is the smallest possible incremental cost to get a real dashboard onto a system that is already in production. Point your OpenAI or Anthropic client at oai.helicone.ai/v1 (or the Anthropic equivalent), add a Helicone-Auth header, ship. Now cost-per-user, p95 latency, error rates, prompt-level slow queries, and cache-hit ratio land in a dashboard that day. Caching, retry, rate limit per user/key, PII filtering, and the AI Gateway's cross-provider routing all sit at the same layer — they are gateway features, not observability features, but they share the proxy. The trade-off is the inverse of LangSmith's: you give up framework-awareness (no node spans, no prompt-version linkage) and get back zero migration cost. Replay and experiments let you re-run a captured prompt with a new model or new template, scored by an evaluator, but the gravitational centre is "what is my prod doing right now."

Integration shape — gateway proxy first, OTel optional

The proxy is the wire-shape, full stop. Async logging via SDK is supported for cases where you cannot reroute traffic, and the project does emit and accept some OTel signals, but the canonical install is "change one URL." For teams whose cost, quality, latency conversation is happening at the platform-bills level — finance asks "where did the $40k go," ops asks "why did the 99th percentile blow up at 14:00" — Helicone is the lowest-friction way to answer those without changing application code.

Arize Phoenix — deep dive

Data model — OpenInference spans on top of OpenTelemetry

Phoenix does not invent a proprietary trace schema. The data model is OpenInference, an open spec layered on OpenTelemetry that defines span attributes for LLMs, retrievers, embeddings, tool calls, and agent steps. The instrumentors are OTel auto-instrumentors — point an OTel SDK at Phoenix's OTLP endpoint and a LangChain, LlamaIndex, OpenAI SDK, Anthropic SDK, or Bedrock call shows up as a structured LLM span without you writing wrappers. The implication is portability: the same instrumented app can ship spans to Phoenix today, to Jaeger or Tempo or Datadog tomorrow, without re-instrumenting. That property does not exist in the other three.

The loop it closes — OTel-native monitoring with ML-obs drift

Phoenix inherits the ML-observability tradition from its parent Arize. That means drift is not an afterthought: embedding-similarity drift on RAG inputs, retrieval-quality drift over time, UMAP clusters of model outputs, and per-cohort regression are first-class views — the kind of monitoring ML teams have done on tabular models for a decade, ported to the LLM era. phoenix.evals is a Python library for running offline evaluators (LLM-judge, hallucination, retrieval quality, custom) over stored traces, and it is comfortable in a notebook. The loop closed is the "monitor in prod, notice drift, run a notebook eval, file a ticket" loop, not the "block the PR" loop. For RAG specifically, the embedding-drift view is the one piece Evaluating RAG describes as hard to bolt on later.

Integration shape — OpenTelemetry, OSS, self-host first

Arize Phoenix is the most genuinely open of the four: Elastic-2.0 licensed, runs from pip install arize-phoenix in a notebook, scales to a Docker/Kubernetes deployment, with Phoenix Cloud as a hosted convenience and Arize AX as the paid production tier when you outgrow the OSS server. The OTel-native design is the lock-in story in reverse — your instrumentation is portable by construction, and Phoenix's value lands on top of that portability rather than under it. The trade-off is workflow density: Phoenix gives you the primitives but it does not push you toward a CI workflow the way Braintrust does or a Prompt Hub workflow the way LangSmith does. You assemble the loop yourself.

Cross-cutting comparison

Instrumentation shape — SDK vs proxy vs OTel

Four products, three architectures for how the trace data leaves your process. LangSmith and Braintrust both hand you an SDK, but they sit on opposite ends of "auto vs explicit": LangSmith is framework-coupled (LangChain's tracer callback fires for free; the @traceable decorator covers the rest), while Braintrust is wrapper-coupled (you opt in with wrapOpenAI or @traced, and the win is that an eval is just another function with those wrappers). Helicone takes the opposite path entirely: the proxy sits in front of every request, so instrumentation is a base_url change rather than a code change, and the trade-off is that you only see what crosses the wire. Phoenix moves outside the proprietary-SDK frame altogether by speaking OpenTelemetry/OpenInference, which is the only one of the four where the instrumentation outlives the vendor — re-point the OTel exporter and the same spans go anywhere. If your team standardises on OTel for the rest of the stack, that asymmetry is decisive.

Evaluation model — offline vs online vs CI

All four can run an LLM-judge against a dataset. They diverge on which moment of the lifecycle the eval is meant to live in. Braintrust is the strongest opinion: an eval is code, it runs on every PR, and a regression blocks merge — the answer to "did this get worse" arrives before the change ships. LangSmith straddles dev and prod through the Prompt Hub plus online evaluators that sample production traces and surface regressions after the fact. Helicone anchors its evals to captured production requests — its replay/experiments flow re-runs a real prod prompt with a new template, which is closer to a post-hoc what-if than a gating test. Phoenix runs evaluators as a Python library over OTel traces (comfortable in a notebook, less so as a CI gate) and adds the embedding-drift view as a continuous monitor rather than a discrete test. None of these is wrong; they answer the question at different points, and the right pick depends on whether your bug is "we regressed on a known dataset" (Braintrust), "production looks weird and I want to know why" (Helicone, Phoenix), or "the prompt change I'm about to ship is risky" (LangSmith).

Open-source vs SaaS — and who holds the data

This axis splits cleanly into two pairs. LangSmith and Braintrust are SaaS-first products with paid self-host tiers for enterprises that cannot send prompts to a vendor — the OSS components are SDKs, not the backend that stores your data. Helicone and Phoenix are genuinely open source: Helicone's gateway is Apache-2.0 and you can run the full stack on your own boxes; Phoenix is Elastic-2.0 and is designed to run anywhere, from a notebook cell to a Kubernetes deployment. For a regulated workload — health, finance, government — that pair is the natural starting point, with Phoenix's OTel-native design and Helicone's gateway shape covering different halves of the problem. For everyone else the trade is convenience: SaaS-first products move faster on UI polish and online-eval features; OSS products move faster on portability and "we own the data plane."

When to pick which

Further reading

On this wiki:

• Evals 101 — what an eval set actually contains, why LLM-judges are not free, and the minimum-viable harness before any platform.
• Agent Frameworks — what a framework adds over raw model calls, since LangSmith's value is tightest when your app is already framework-shaped.
• Cost, Quality, Latency — the three-way trade these platforms instrument, and why gateway-level numbers (Helicone's home turf) and quality scores (Braintrust's) are not interchangeable.
• Evaluating RAG — the retrieval/grounding/answer-quality split, useful before you pick a platform to score it on.
• Evaluating Memory Quality — memory-specific metrics (recall@k, staleness, drift) that are easier to monitor on a platform that already speaks embeddings.

Project sources:

• LangSmith docs — runs, datasets, Prompt Hub, evaluators, and LangSmith Deployment.
• Braintrust docs — Eval() as code, experiments, autoevals, online logging, playground.
• Helicone docs — gateway proxy, caching, rate limits, sessions, AI Gateway routing (source at github.com/Helicone/helicone).
• Arize Phoenix docs — OpenInference, OTel instrumentation, phoenix.evals, drift views (source at github.com/Arize-ai/phoenix).