Langfuse AI Gateway is a proxy, in public beta since November 9, 2026, that sits between your apps or coding agents and model providers such as OpenAI and Anthropic. You change only the base URL and the API key, and every call is recorded in Langfuse as a generation with model, tokens, cost and latency, plus the full prompt and response if you opt in. It is MIT licensed, free to self host, and on Langfuse Cloud each recorded call counts as a billable unit.
That makes it different from most gateways, which stop at counting tokens. This guide covers what the Langfuse gateway records, how the keys and routing work, what it costs on Cloud and self hosted, the beta limits you should check before trusting its numbers, and when you should keep using the Langfuse SDK instead. It is written from an observability angle. If you are choosing between gateways and routers in general, the drydock.si guide to LLM gateways versus OpenRouter covers that ground.
What Langfuse AI Gateway is
Langfuse built its name on open source tracing, prompt management and evals. The gateway adds a central entry point for model traffic. Langfuse's launch post frames the problem plainly: companies now run coding agents on every laptop, assistants in every team and many production apps, and finance and platform teams cannot see who uses which model, for what, at what cost.
SDK instrumentation only covers code you control. A coding agent such as Claude Code or Codex, or a script someone wrote last month, never sends a trace unless something sits in the path of its model calls. A gateway does. Each request passes through it, so nothing is missed, and because Langfuse stores the call as a normal trace, you get search, evaluations and annotation on top of the cost line.
The gateway is written in Rust and ships as a separate container, langfuse/ai-gateway. Langfuse describes it as a single 22 MB Docker image and claims under 3 milliseconds median added latency with tracing on. Those are vendor figures. On November 12 we saw the 4.79.0 image on Docker Hub listed at 26.5 MB compressed for amd64 and 24.9 MB for arm64, close to the claim.
How a request flows through the gateway
The concepts page describes four steps. Your client sends a normal provider request to the gateway with a gateway key. The gateway checks the key and maps the model name to one of your organization's provider connections. It calls the provider with the real provider key and streams the answer back. After the response, it records the call in Langfuse asynchronously.
Three ideas make it work:
- Three kinds of keys. Provider API keys live only in the gateway. People get user keys, which stop working when they leave the organization. CI jobs and shared services get service account keys. You can rotate an OpenAI or Anthropic key without touching a single client, and revoke one person without affecting anyone else.
- The URL path picks the API format.
/openai/v1/responses,/openai/v1/chat/completions,/openai/v1/decisionsand/openai/v1/embeddingstake OpenAI formats, and/anthropic/v1/messagestakes Anthropic's. The gateway does not translate between formats, so an OpenAI style client can only reach providers that speak the OpenAI API. - The model ID picks the connection. IDs take the form
<connection>:<model>, so a connection namedopenai-prodservesopenai-prod:models. Coding agents that send plain provider model names still work for OpenAI and Anthropic.
Our own check matched the docs on one detail: the gateway answers in the provider's native format even when it rejects you. An invalid key on the OpenAI route returned an OpenAI style invalid_api_key error, and on the Anthropic route an Anthropic style authentication_error, both with a request_id you can quote to support. That is why SDKs and agents treat it like the real provider.
What the gateway records: FULL, USAGE and NONE
The tracing docs list three ingestion modes, set once for the whole organization:
| Mode | What is stored | Where |
|---|---|---|
| USAGE (default) | Model, token usage including cached and reasoning tokens, cost, latency, time to first token, errors, session, user and key metadata | Cloud and self hosted |
| FULL | Everything in USAGE plus the full input (system prompt, tools, messages) and the streamed output, including reasoning and tool calls | Cloud and self hosted |
| NONE | Nothing. The gateway only routes requests | Self hosted only |
USAGE is the safe default when prompts must not land in a third party database. FULL is what turns the gateway into an evaluation feed: with real prompts and responses stored, you can run LLM as a judge evaluators on production traffic, send calls to annotation queues, or build datasets from real usage.
You can enrich traces in either mode with optional request headers for user, session, environment, tags, metadata and trace name. For supported coding agents the gateway also infers the agent's own session ID, so a Claude Code session shows up as one Langfuse session without extra work.
Combining the gateway with SDK tracing
The gateway only sees model calls. It cannot see retrieval, tool execution or the business logic around them. If you already trace with the Langfuse SDK or OpenTelemetry, send the W3C traceparent header and the gateway nests its generation inside your existing trace. Langfuse warns against also tracing the same call with a provider SDK integration, because the call would then be recorded twice. If you are still deciding how to instrument in the first place, our comparison of OpenLLMetry and Langfuse explains the capture library versus platform split.
Langfuse gateway pricing on Cloud and self hosted
Self hosting costs nothing in license fees. The core gateway is MIT licensed, like the rest of the Langfuse core, which our guide on whether Langfuse is free for commercial use explains in detail. You still pay for the servers.
On Langfuse Cloud there is no token markup. The gateway calls providers with your own keys, so OpenAI or Anthropic bill you directly under your existing contract. What Langfuse charges for is storage. Its billing FAQ says each generation the gateway records counts as a billable unit, the same as any other observation.
One point needs care. The launch post says Cloud charges only if you decide to log requests and that non logged requests are free. But both ingestion modes available on Cloud (USAGE and FULL) record a generation, NONE is self hosted only, and the v4.73.0 release notes say the gateway now always records requests on Cloud. Read together, plan for one billable unit per call on Cloud, and check your usage page after a test week.
Current Cloud plans from the Langfuse pricing page, checked November 12:
| Plan | Monthly price | Included units | Extra units | Gateway rate limit (beta) |
|---|---|---|---|---|
| Hobby | Free | 50,000 | Not available | 100 requests a minute, 64 MiB a minute |
| Core | 29 dollars | 100,000 | 8 dollars per 100,000 | 500 requests a minute, 256 MiB a minute |
| Pro | 199 dollars | 100,000 | 8 dollars per 100,000 | 10,000 requests a minute, 5 GiB a minute |
| Enterprise | 2,499 dollars | 100,000 | 8 dollars per 100,000 | 10,000 requests a minute, 5 GiB a minute |
A busy coding agent can make hundreds of model calls in a session, so the Hobby allowance and its 100 requests a minute limit suit a trial, not a team rollout. Over the limit, the gateway returns a 429 with a Retry After header in the provider's format, so clients back off normally.
Supported providers and clients
The gateway launched with OpenAI and Anthropic. Amazon Bedrock followed in v4.77.0 on November 9. The v4.79.0 release on November 12 made OpenRouter, xAI and Vercel AI Gateway generally available, made custom base URLs generally available, and routed the OpenAI Decisions API as its own format. Then v4.80.0 on November 13 added Google AI Studio (OpenAI Chat Completions and Embeddings formats) and Microsoft Azure through Microsoft Foundry, and renamed the gateway settings area to Connections & Models. The models page lists which API formats each provider accepts. Vertex is not on that list yet.
On the client side, Langfuse publishes setup steps for Claude Code, Claude Desktop, Codex, GitHub Copilot, OpenCode, Pi, the OpenAI and Anthropic SDKs and the Vercel AI SDK. Claude Code, for example, needs version 2.1 or later. You log out of any Claude subscription, set two environment variables (the gateway's Anthropic URL and the gateway key as the auth token) and unset any Anthropic API key, which would otherwise cause a 401.
Cloud hosts are regional: EU, US, Japan and HIPAA each have their own gateway host, and a key only works on its own region's host.
Beta limits to check before you trust the numbers
Langfuse's known limitations page is unusually candid. The items that matter most for observability and cost reporting:
- Cost bugs. OpenAI cache write tokens are priced at zero, and Anthropic one hour cache writes are priced at the five minute rate, 37.5% too low. Fixes are in progress. Until then, gateway cost dashboards can understate spend for cache heavy agents.
- Large payloads. Inputs over 5 MiB are not recorded (usage and cost still are), and outputs over 1 MiB are recorded only partially. Long Codex and Claude Code sessions can hit the input limit.
- One trace per model call for Claude Code, OpenCode and Pi unless you set up turn grouping. Only Codex groups a turn into one trace by default.
- No retries, fallbacks or caching, no websockets or realtime APIs, and hard limits of 16 MiB per request body and 600 seconds per call.
- Spend limits are roadmap only. The docs list hard spend caps, provider fallbacks, guardrails and PII redaction as planned, not available.
None of these is unusual for a three day old beta, but they decide whether the gateway can be your source of truth for finance reporting today. For agent teams, compare the trace shape it produces with what our AI agent observability guide says you need: tool calls and permissions around the model call still come from SDK or OpenTelemetry instrumentation.
Self hosting the gateway
On self hosted Langfuse the gateway is a separate, stateless container deployed with the Helm chart or Docker Compose next to your existing instance. The self hosting guide sets the requirements: Langfuse v4.75.0 or later with matching versions for web and gateway, the v4 data model, a DNS name with TLS terminated at your proxy, outbound HTTPS to your providers, and an encryption key on web for stored provider keys. The gateway itself holds no database credentials. It talks only to Langfuse web and the providers.
Self hosting brings two real advantages: the NONE mode for teams that only want to hand out gateway keys, and no Cloud rate limits. Langfuse suggests at least two replicas in production and warns that every request also hits Langfuse web to resolve the key (cached for 60 seconds), so plan capacity for both.
Who should use the Langfuse gateway
It fits well if you already run Langfuse and want coding agent usage and cost per person or team without asking anyone to change code, or if you want a central place to hold provider keys. It also fits teams that want to evaluate real production prompts but cannot instrument every app.
It fits less well if you need fallbacks, caching or spend caps today, or if your reporting depends on exact cache costs before the pricing bugs are fixed. Teams already on LangSmith should note that LangSmith also has a model gateway; our Langfuse vs LangSmith comparison weighs the two platforms. Helicone, the older proxy first option, is maintained but no longer gets new features, as covered in our roundup of the best LLM observability tools. If you are weighing a move off it, read our Helicone vs Langfuse comparison.
FAQ
Is Langfuse AI Gateway open source?
Yes. Langfuse says the core gateway is MIT licensed, like the rest of the Langfuse core, and you can run it on your own infrastructure or use the managed version on Langfuse Cloud.
Does Langfuse store my prompts when I use the gateway?
Not by default. The default USAGE mode stores the model, tokens, cost, latency and metadata without the input or output. Prompts and responses are stored only if an admin switches the organization to FULL mode. Self hosted deployments can also use NONE, which records nothing.
Do I need to change my code to use it?
No SDK changes are needed. You change the base URL to your region's gateway host and replace the provider key with a gateway key. Requests and responses stay in the provider's native format.
How much does the Langfuse gateway cost?
Self hosting is free apart from your infrastructure. On Langfuse Cloud there is no token markup, and each recorded call counts as a billable unit: Hobby includes 50,000 units a month free, and Core (29 dollars a month) and Pro (199 dollars a month) include 100,000 units with extra units at 8 dollars per 100,000.
Which providers does the gateway support?
As of November 13, 2026: OpenAI, Anthropic, Amazon Bedrock, Google AI Studio, Microsoft Azure (Microsoft Foundry), OpenRouter, xAI and Vercel AI Gateway, plus custom base URLs. Vertex is not listed yet.
Can I use the gateway and the Langfuse SDK together?
Yes. Send the W3C traceparent header from your current span and the gateway records its generation inside your trace. Do not also trace the same call with a provider SDK integration, or it will be recorded twice.
