// category · 10 tools · 5 open source
LLM Observability Tools
Tracing, logging and monitoring for LLM apps and agents in production, so you can see what happened, what it cost and where quality slipped.
LLM observability tools record what a language model application does in production so you can debug it, measure cost and quality, and catch regressions. This category covers tracing platforms, instrumentation libraries and monitoring products for single model calls, retrieval pipelines and multistep agents.
At minimum, observability captures each prompt and completion, the model and parameters used, latency, token counts and cost. For retrieval apps it adds the documents fetched. For agents it adds the whole loop: which tools were offered and chosen, their arguments and results, retries, subagents and how many steps a task took. Quality signals such as user feedback and automated scores are attached to the same trace, which links monitoring to evaluation.
Open source platforms. Langfuse has an MIT licensed core and became part of ClickHouse in January 2026. Arize Phoenix is free to self host under the Elastic License 2.0, and Arize became part of Dynatrace on October 1, 2026. Opik and Laminar are Apache 2.0 options, with Laminar aimed at long running agents.
Instrumentation. OpenLLMetry is not a dashboard but a set of OpenTelemetry instrumentations that send traces to any compatible backend. OpenTelemetry's GenAI conventions are becoming the shared format, which keeps your code portable between vendors.
Hosted products. LangSmith fits teams already on LangChain and LangGraph. Helicone, HoneyHive, Galileo, W&B Weave and Braintrust cover hosted tracing, monitoring and evaluation with different emphases.
Agents raised the stakes in 2026. In October, Partnership on AI reported six telemetry gaps in widely used agent frameworks, including missing records of permission changes, memory changes and human approvals. A tracing platform cannot show data that the framework never emitted, so check what your stack actually records.
To choose, start with our roundup of the best LLM observability tools, then read AI agent observability for agent specific monitoring. Head to head comparisons: Langfuse vs LangSmith and Langfuse vs Arize Phoenix.
// guides
Guides to LLM observability tools
Best LLM Observability Tools in 2026, Compared
The best LLM observability tools in 2026 compared: Langfuse, LangSmith, Phoenix, Opik, Weave and more, with licenses, self hosting and who now owns what.
AI Agent Observability: What It Is and Top Tools
AI agent observability explained: trace tool loops, permissions and memory, the six PAI telemetry gaps, and which tools fit, from Langfuse to Datadog.
HoneyHive vs Langfuse: Free Tiers, Evals and Hosting
HoneyHive vs Langfuse in 2026: what each one bills as an event, free tier math from a real SDK test, evals, self hosting and which team should pick which.
Helicone vs Langfuse: Which to Pick in 2026
Helicone vs Langfuse in 2026: a proxy first logger in maintenance mode against an actively built tracing and eval platform. Pricing, gateways and when to move.
OpenLLMetry vs Langfuse: Instrumentation or Platform?
OpenLLMetry vs Langfuse in 2026: one captures LLM traces, the other stores and scores them. See how they fit together, what each costs and when to pick which.
Is LangSmith API Free? Keys, Limits and Costs
Is LangSmith API free? Yes: API keys and the REST API cost nothing, and the Developer plan covers 5,000 traces a month. See key types, limits and overage costs.
Is Langfuse Free for Commercial Use? (2026)
Is Langfuse free for commercial use? Yes: the MIT core can be self hosted in paid products at no cost. See what needs a license key and when the cloud bills.
Datadog LLM Observability vs Langfuse (2026)
Datadog LLM Observability vs Langfuse: how each bills (LLM spans vs units), retention, self hosting, evals and setup, with worked cost examples for 2026.
Langfuse vs LangSmith: Which Should You Use?
Langfuse vs LangSmith in 2026: open source license, self hosting, pricing, retention, framework fit and migration tips, so you can pick the right LLM tracer.
Langfuse vs Arize Phoenix: Which to Self Host?
Langfuse vs Arize Phoenix in 2026: licenses, self hosting stacks, features, managed pricing, new owners and which open LLM tracing tool fits your team best.
Langfuse AI Gateway: What It Logs and Costs
Langfuse AI Gateway routes OpenAI and Anthropic calls through Langfuse and traces each one. What it records, what it costs, its beta limits and who it suits.
How to Use LangSmith with LangGraph (2026)
How to use LangSmith with LangGraph: set four environment variables, read node traces and threads, then score your graph and single nodes on a dataset.
Helicone vs LangSmith: Proxy Logs or Agent Traces?
Helicone vs LangSmith in 2026: request logs and a mature gateway against agent traces, evals and a new beta gateway. What each bills, sees and costs, tested.
Patronus AI Agent Failure Research, Explained
Patronus AI agent failure research in plain English: what the 63% figure really means, what the TRAIL benchmark found and how to find failures in your own agents.
LLM Evaluation for A/B Testing: How to Run It
LLM evaluation for A/B testing explained: when offline evals stop being enough, what live metrics to log, how big the test must be, and traps that skew results.
LLM as a Judge vs Agent as a Judge: When to Switch
LLM as a judge vs agent as a judge: what each grades, what DevAI, AJ-Bench and Mind2Web 2 found, what an agent judge costs, and when to switch.
Ragas vs Langfuse for RAG Evaluation (2026)
Ragas vs Langfuse is a metrics library versus an open source tracing platform. How scores attach to traces, when to use both, pricing, and a tested wiring note.
Ragas vs LangSmith for RAG Evaluation (2026)
Ragas vs LangSmith: a free metrics library versus a paid eval platform. What each scores, what it costs, a tested way to run Ragas inside LangSmith, and Ragas risks.
RAG Evaluation Through Python: A Tested Walkthrough
RAG evaluation through Python, tested: score retrieval with plain Python hit rate, recall and MRR, add Ragas 0.4 metrics, and see which answer scores mislead.
LLM as a Judge: How It Works and When to Trust It
LLM as a judge explained: how model graders work, the biases that skew them, how to validate one against human labels, and which tools to use.
How to Evaluate LLM Applications (2026 Guide)
How to evaluate LLM applications step by step: error analysis, code checks, calibrated LLM judges, CI gates and production sampling, with tools for each stage.
RAG Evaluation Metrics: Ragas vs DeepEval vs TruLens
RAG evaluation metrics explained: faithfulness, relevancy, context precision and recall, which need a reference, and how Ragas, DeepEval and TruLens differ.
LLM Judge Model: Frontier, Small or Decision Model?
Which LLM judge model should grade your outputs? Frontier LLMs, small judges and decision models like Jev compared on accuracy, cost and calibration.
How to Evaluate LLMs Before Production
How to evaluate LLMs before production: set the product decision, build a production like offline suite, wire CI gates, then open shadow or A/B traffic.