LLM evaluation for A/B testing means splitting real users between two versions of an AI feature (a new prompt, a new model or a new retrieval setup) and measuring which one does better on outcomes you decided in advance: user behavior, cost, latency, safety filters and sampled quality grades. Run offline evals first to catch obvious regressions, then a shadow test to compare both versions on the same live inputs without exposing anyone, and only then a true A/B test, because only real users can tell you whether the change made the product more useful. Plan for thousands of users per variant, check that traffic actually split the way you configured it, and never ship on an LLM judge score alone.
Offline evals vs A/B tests: what each one can tell you
An offline eval runs a fixed dataset through your app and scores the outputs, usually in CI. It is fast, cheap and repeatable, and it is the right tool for catching regressions before anyone sees them. Our playbook on how to evaluate LLM applications covers that loop step by step.
What an offline eval cannot do is measure whether people get more value. Microsoft's Experimentation Platform team, which helped product groups launch LLM features, put it plainly in its metric framework for LLM features: offline evaluation suits early development, but it "cannot assess how model changes benefit or degrade the user experience in production." The same article calls A/B testing "the golden standard to causally measure the impact of any change to the product."
Offline evals decide what is safe to try. A/B tests decide what to keep.
| Question | Offline eval | Shadow test | A/B test |
|---|---|---|---|
| Did a known failure come back? | Yes | Partly | Too slow |
| Is the new version cheaper or faster on real inputs? | Roughly | Yes | Yes |
| Do users accept, keep or return more? | No | No | Yes |
| Risk to users | None | None | Some |
The four experiments Microsoft runs for LLM features
The ExP team describes four experiment types, two for launch and two for later changes.
1. Dark mode experiment. The feature's components load, but nothing is shown to the user. You learn whether the system holds up under real load (errors, rate limits, latency) before anyone depends on it.
2. Zero to one experiment. Treatment gets the LLM feature and control does not. This answers "should this feature exist at all?" Roll it out gradually so you know you have enough GPU or API capacity, and watch the product's main success metric and its guardrail metrics alongside the new productivity metrics.
3. Shadow experiment. Every user's input goes to both the control workflow and the treatment workflow, but users only ever see the control response. Because both versions answer exactly the same requests from exactly the same users, the comparison has much less noise than a normal split, and ExP notes you can get more sensitivity again with paired t tests. The catch: anything that depends on the user's reaction (accepts, edits, thumbs, retention) cannot be measured, because nobody saw the treatment.
4. One to N experiment. The ordinary A/B test: some users get the new prompt, model or interface, the rest keep the current one.
For most prompt and model changes after launch, the sequence is shadow first (is it cheaper, faster, safer and graded better on live inputs?), then a one to N test (do users actually benefit?).
What to measure in an LLM A/B test
A good LLM experiment carries four layers of metrics. Pick only one and you will ship something that wins there and loses elsewhere.
Cost and capacity
Log prompt tokens, completion tokens and their total for every call. ExP treats total tokens as the main utilization metric, and also tracks the number of 429 (rate limited) responses at the 90th or 95th percentile, calls cut off at the token limit (a finish_reason of length), and "wasted" completion tokens spent on responses that gave the user nothing useful. A variant that writes twice as much text may win on engagement and still lose on cost per useful answer.
Speed
Measure time to first token at several percentiles, requests per second and streamed tokens per second. If retrieval or tool calls sit between the model and the screen, time each stage separately.
Safety and filtering
Count the share of prompts and responses stopped by your content filter. A new system prompt that trips the filter more often looks like a quality drop to users.
User value and quality
This is the layer that justifies the test. ExP builds a funnel: how often the model was called, how often a response was shown, how often the user accepted it, and how much of it they kept. Around that sit quality signals such as prompt and response length, the edit distance between a user's consecutive prompts (rephrasing the same request suggests the first answer missed), edit distance between the response and what the user kept, thumbs up and down, and retention on the feature over days, which also exposes novelty effects.
Treat thumbs as a weak signal. The ExP article warns that explicit feedback suffers from small samples and selection bias, because the people who click thumbs are not representative of all users.
How to run an A/B test on an LLM prompt or model, step by step
- Write the decision down first. Name the change, the one primary metric that decides the test (for example "share of sessions where the user accepted an answer"), the guardrails that must not get worse (cost per session, 95th percentile latency, filter rate) and the minimum effect worth shipping.
- Pass the offline gate. The candidate must clear your CI eval set. Our guide to LLM evaluation metrics helps pick the checks.
- Randomize by user, not by request. If the same person sees prompt A on one message and prompt B on the next, a single conversation mixes both and you cannot attribute the outcome. Feature flag tools assign a user to one variant and keep them there.
- Tag every trace with the variant. Store the variant, prompt version and model id on each trace so every metric can be grouped by variant. An observability tool such as Langfuse, LangSmith or Arize Phoenix makes this a metadata field rather than a project.
- Shadow it if you can afford double calls. Run the candidate on the same live inputs, show users only the control and grade both outputs.
- Start small, then widen. Begin with a few percent of traffic, check guardrails and the sample ratio, then move to the split your sample size calculation needs.
- Run for full weeks. Weekday and weekend users behave differently.
- Decide on the primary metric with its confidence interval, check the guardrails, then read the judge grades and a sample of transcripts to understand why the winner won.
How many users an LLM A/B test needs
LLM features rarely move metrics by much, and small effects need large samples. Using the standard two proportion formula (5% significance, 80% power), we computed these sizes in Python:
- To detect an acceptance rate moving from 20% to 22%, you need about 6,500 users in each variant.
- To detect 20% to 21%, you need about 25,600 in each variant.
- To detect 60% to 63%, you need about 4,100 in each variant.
Halving the effect you want to detect roughly quadruples the sample. With a few hundred active users a week, a small prompt tweak will not get a clear answer; use shadow tests and offline evals for those, and save A/B tests for bigger changes.
Model outputs also vary on their own. Thinking Machines Lab sampled 1,000 completions of the same prompt at temperature zero from Qwen3-235B and got 80 different completions (Defeating Nondeterminism in LLM Inference), because server load changes batch sizes and batch size changes the arithmetic. That noise is one more reason to randomize by user and measure over many sessions rather than compare a handful of side by side outputs.
Check the split before you read the results
A sample ratio mismatch (SRM) happens when the share of users in each variant differs from what you configured. In a KDD 2019 paper, researchers from Microsoft, Booking.com and Outreach.io report that about 6% of Microsoft experiments showed an SRM, and that an SRM in most cases invalidates the result entirely. Their example: a 50.2 to 49.8 split (821,588 versus 815,482 users) on a planned 50/50 test. A chi square test on those counts gives a p value of about 0.0000018, which matches their statement that the odds of chance are below 1 in 500,000.
LLM features create new ways to break the split. A variant that times out more often may drop users before they are logged, and one that trips the content filter may never record a "response shown" event. Run the chi square check daily, and if it fails, find the cause before you read any other metric.
LLM evaluation without ground truth on live traffic
Live traffic has no reference answers, so automated quality grades on an A/B test come from reference free checks: code checks (valid JSON, banned content, length limits) and LLM judges scoring a sample of outputs against a rubric. Statsig's online evals documentation describes exactly this pattern: the live prompt serves users, candidate prompts run in the background on the same input, and graders (LLM as a judge, string comparison, text similarity or Python) score both on a 0 to 1 scale, with a sampling rate setting for how much traffic gets graded. The feature is labeled Early Access in its docs.
Judge grades help explain a result and are the main signal in shadow tests. They should not decide a one to N test: a judge can prefer the longer answer while users prefer the short one, and it can drift when its provider updates the model. Validate the judge against human labels first, as our LLM as a judge guide explains, and keep the user outcome as the deciding metric.
Tools that support A/B testing LLM prompts
A feature flag, a variant tag on each trace and a notebook are enough to start. Three documented options help further.
Langfuse prompt labels. The Langfuse A/B testing docs show labeling two prompt versions (for example prod-a and prod-b), picking one at random in code and linking it to the generation. The Prompts view then compares versions on latency, tokens, cost and evaluation scores. The random choice in the example happens per call, so for user level experiments, assign the label from a hash of the user id instead. Langfuse suggests it for apps with clear success measures and varied inputs, or as a canary after dataset testing.
Statsig prompts and online evals. Covered above: versioned prompts you can change without a deploy, plus shadow grading of candidate prompts on live inputs.
LaunchDarkly AgentControl. LaunchDarkly's AgentControl experimentation docs separate the two jobs clearly: a config's Monitoring tab shows tokens and call duration per variation, while an experiment measures end user behavior such as clicks. A guarded rollout and an experiment cannot run on the same flag at once.
For the offline side before a test, hosted eval platforms such as Braintrust and the frameworks in our LLM evaluation frameworks comparison run the dataset experiments that decide which candidates deserve live traffic.
Common mistakes in LLM A/B tests
- Shipping on a judge score. A higher rubric grade with flat user metrics is not a win. Check whether the judge simply likes longer answers.
- Ignoring cost. A variant that lifts acceptance by one point and doubles tokens may lose money. Report cost per accepted answer next to the primary metric.
- Per request randomization. It mixes variants inside one conversation and inflates apparent significance.
- Peeking daily and stopping on the first significant day. Fix the sample size or use a sequential method your tool supports.
- Changing the prompt or model version mid test. Any edit starts a new experiment, so pin model versions where the API allows it.
FAQ
Can you A/B test LLM prompts?
Yes. Assign each user to one prompt version with a feature flag or a prompt management tool, tag every trace with the version, and compare a user outcome metric such as acceptance rate, plus cost, latency and filter rate as guardrails.
What is the difference between offline and online LLM evaluation?
Offline evaluation scores a fixed dataset before release, usually in CI, to catch regressions. Online evaluation scores live production traffic, either by grading sampled outputs or by running a controlled experiment where real users see different versions.
What is a shadow experiment for an LLM?
A shadow experiment sends each live input to both the current and the new version but shows users only the current response. It compares cost, latency, safety and graded quality on identical traffic with no user risk, but it cannot measure user reactions.
How do you evaluate an LLM without ground truth?
Use reference free checks: code assertions for format and policy, LLM judges validated against human labels, and user behavior such as acceptance, edits, rephrasing and retention. In an A/B test the user behavior metrics carry the decision.
How long should an LLM A/B test run?
Until it reaches the sample size you calculated before starting, and at least one full week so weekday and weekend use are both included. Keep watching retention after launch for novelty effects.
