To evaluate LLMs before production, treat offline evaluation as a go or no go gate, not a demo. Write down the product decision and the safety and cost guardrails first, build a versioned dataset that looks like real traffic (including hard and adversarial cases), compare every candidate against the current baseline with one major change at a time, and fail CI when a release critical check regresses. Only after that gate is green should you open shadow traffic or a controlled A/B test. Offline evals do not prove users will love the change; they prove the change is safe enough to try.

This guide is the pre production checklist. For the day to day error analysis loop inside an app, see how to evaluate LLM applications. For live experiments after the offline gate, see LLM evaluation for A/B testing. For picking a model rather than shipping a feature, see how to evaluate LLM model performance. Facts below were checked against primary docs on Nov 6, 2026. We did not ship a production LLM feature for this guide.

What "before production" means

"Before production" is the moment a prompt, model, retrieval index, tool schema or agent workflow is about to leave the lab. The question is no longer "can the model do this on a clean demo set?" It is "do we have enough evidence to expose real users, and which mistakes are unacceptable?"

GitHub's engineering blog makes that shift concrete in How to evaluate LLMs before production (August 25, 2026), written from work on LLM assisted secret scanning. Their product decision was not "classify this string correctly." It was "can we cut false positive alerts while keeping recall inside a safety band, and inside latency and cost limits?" Precision was the primary outcome. Recall was a safety constraint. Latency, cost, reliability and compatibility were operational guardrails. A change that looked great on precision but broke the recall band did not advance.

That three layer split (primary outcome, safety constraint, operational guardrails) is the first thing to write down. Without it, every metric looks equally important and every experiment looks like a win.

The pre production evaluation stack

LangChain's How to Test LLMs and AI Agents (October 27, 2026) separates an eval (a measurement) from a test (a pass or fail rule that can block release). Arize's EvalOps guide (last reviewed September 22, 2026) adds the release policy: SHIP, REVIEW or HOLD, based on versioned evidence. Put together, a useful pre production stack looks like this:

LayerWhat it answersTypical check
Product decisionWhat are we trying to improve, and what must not get worse?One primary metric plus written guardrails
DatasetDo the cases look like production?Versioned goldens with hard and past failure cases
Deterministic checksDid format, schema, citations or tool args break?Code assertions in CI
Judgment checksIs the answer good enough on fuzzy criteria?Calibrated LLM judge or human review
Trajectory checksDid the agent take a safe path?Required or forbidden tool steps
Baseline compareDid we regress vs what is live today?Same dataset, candidate vs current
Release policyShip, hold, or send to a human?Thresholds that block CI
RollbackCan we turn it off fast?Kill switch and non AI fallback

Skip any one of these and you usually discover the gap after users do.

Step 1: Write the product decision and guardrails

Copy GitHub's habit. One sentence for the decision. One primary metric. Explicit safety constraints. Explicit operational guardrails.

Examples:

  • Support bot: raise first contact resolution; never invent a refund; p95 latency under 4 seconds; cost per session under budget.
  • RAG assistant: raise grounded answer rate; never cite a source that was not retrieved; context window and token cost stay flat.
  • Agent with tools: raise task completion; zero unauthorized writes; every money moving tool needs an approval step.

If two metrics conflict, name which one wins. GitHub's hypothetical table is useful here: Experiment A with a large precision gain and a recall breach does not ship. Experiment B with a moderate precision gain inside the recall band continues.

Step 2: Build a production like offline dataset

An offline set that is cleaner than production will flatter every candidate. GitHub's secret scanning work kept surrounding code, incomplete context and distracting nearby strings in the examples, because those distractions are where real failures live. LangChain's testing guide says the same thing in dataset form: start with common user tasks, known production failures, high risk edge cases and adversarial requests, then grow the set as live traces reveal new bugs.

For each case, store at least:

  1. The input (and any prior thread state the component needs).
  2. The expected outcome or the rule an evaluator can check.
  3. Trajectory rules when path matters (must call lookup_policy before answering).
  4. Metadata so you can slice by risk, locale or intent.

Version the dataset. A score without a dataset version cannot be compared to last week's score. When a production failure is confirmed, add it as a regression case before the next release, as both LangChain and Arize recommend.

Public benchmarks and synthetic cases can fill early gaps. They should not replace production like data once you have it.

Step 3: Separate deterministic checks from judgment checks

Do not spend an LLM judge call on something a parser can decide.

Deterministic (cheap, stable, good for every pull request):

  • Valid JSON, schema or enum values
  • Required fields, banned phrases, secret patterns
  • Tool name and argument shape
  • Citations that must appear, or sources that must not be invented
  • Latency and cost thresholds on the eval run itself

Judgment (slower, needs calibration):

  • Helpfulness, tone, completeness
  • Groundedness against retrieved context
  • Policy compliance that depends on context
  • Whether the task was actually finished

For judgment checks, treat the judge like any other model you ship. Our guide to LLM as a judge covers rubrics, human agreement and when not to trust a score. GitHub's triage pattern is practical: auto handle clear low risk cases, route conflicts and high impact cases to humans, and periodically sample high confidence cases so the judge cannot drift unnoticed.

For RAG specifically, libraries such as Ragas give named metrics (faithfulness, answer relevancy, context precision and recall). Some ID based metrics need no model at all. Use them when retrieval quality is part of the release bar; see RAG evaluation metrics for definitions.

Step 4: Compare against the live baseline, one change at a time

Regression testing for LLMs is about deltas, not absolute scores. Run the candidate and the current production configuration on the same versioned dataset. Look at must pass cases and protected slices, not only the average. Arize's EvalOps example shows why: a candidate can rise from 88.4% to 92.6% overall and still HOLD if it records confirmed false successes on a permissions slice the policy forbids.

Change one major variable per experiment when you can (prompt alone, then model alone, then both). GitHub tracked prompt version, model version, precision, recall and latency per run so a win could be attributed. Record the prompt, model, dataset, evaluator and pipeline versions with every result. Without that record, a later reviewer cannot reconstruct why something shipped.

For agents that change external state, also verify the authoritative state after the action. A final answer judge can pass a permissions update that never changed the account. Arize's worked example fails that case even when the tool returned success.

Step 5: Wire the release gate into CI

A checklist that lives in a doc and never fails a build is not a gate. The test types above become release controls when CI runs them and applies thresholds.

Practical wiring with tools already in our catalog:

  • Promptfoo: promptfoo eval exits with code 100 when tests fail (confirmed in our how to use Promptfoo walkthrough). PROMPTFOO_PASS_RATE_THRESHOLD can loosen a 100% bar when you intend to. There is a maintained GitHub Action for before and after evals on pull requests.
  • DeepEval: assert_test inside Pytest, run with deepeval test run, fails the build when a metric misses its threshold. Confident AI's RAG evaluation guide shows a GitHub Actions workflow that runs those tests on push and pull requests.
  • Langfuse: datasets, experiments and scores for offline runs, plus traces once you sample production. Pair it with metric libraries rather than replacing them.
  • Galileo and Evidently: useful when you want hosted monitoring or data quality style reports next to the CI suite; they do not replace a versioned offline gate.

A sensible CI shape for most teams:

  1. Fast deterministic component checks on every pull request.
  2. Offline experiment on the curated regression set before merge or deploy.
  3. Trajectory checks for agents.
  4. Human sign off for high risk or ambiguous slices.
  5. HOLD or unresolved REVIEW blocks deploy; only SHIP continues.

See best LLM evaluation frameworks for tool choice details.

Step 6: Safety, red team and rollback before you flip traffic

Quality gates are not enough for anything that can leak data, call tools or write to customer systems.

Before production:

  • Run injection and jailbreak cases at every untrusted text entry point (user input, retrieved docs, tool results). Our prompt injection attacks guide covers test shapes.
  • Limit tool permissions. Require approval for side effects.
  • Confirm refusal behavior both ways: under refusal (answering when it should decline) and over refusal (blocking valid work).
  • Name an owner, an on call path and a kill switch that can disable the AI feature or fall back to a non AI path within minutes.

Offline evaluation still will not match every production scenario. GitHub's honest framing is the right one: offline work gave them enough structured evidence to justify online experimentation with known risks, not a proof that production would be perfect. Their offline secret scanning work reported a 95% false positive reduction on the evaluated set while keeping recall inside the guardrail. Treat that as a case study of method, not as a number you should expect for your product.

Step 7: What comes after the gate (and what does not)

Once the offline gate is green:

  1. Shadow or canary: run the candidate on live inputs without (or with limited) user exposure.
  2. A/B test: only when you need a causal read on user value. See LLM evaluation for A/B testing.
  3. Online sampling: score a slice of live traces and feed new failures back into the offline set.

Do not skip straight from a notebook demo to full traffic. Do not ship on an uncalibrated judge score alone. Do not treat a public leaderboard win as a production gate for your app.

Who this checklist is for

  • Teams about to put a first LLM feature behind a feature flag
  • Teams that already ship prompts weekly and keep rediscovering the same failures
  • Platform groups that need a written SHIP / REVIEW / HOLD policy auditors can read

If you still need a failure taxonomy, start with how to evaluate LLM applications. For base model bake offs, start with how to evaluate LLM model performance.

FAQ

How do you evaluate LLMs before production?

Define the product decision and guardrails, build a versioned offline dataset that looks like production, run deterministic and calibrated judgment checks against the live baseline, and fail CI when a release critical threshold is missed. Only then open shadow or A/B traffic. GitHub's secret scanning writeup and LangChain's testing guide both follow that order.

Is offline evaluation enough to ship?

No. Offline evaluation shows the candidate is safe enough to try under known conditions. It cannot measure whether users get more value. That needs shadow tests, A/B tests or careful online sampling after the offline gate passes.

What should an LLM release gate block on?

At minimum: critical safety or privacy failures, schema or tool argument breaks that would crash downstream systems, regressions on must pass cases and protected slices, and latency or cost breaches of the written budget. Missing or incomparable evidence should be REVIEW, not a silent pass (Arize EvalOps).

Which tools can run LLM evals in CI?

Promptfoo and DeepEval are common open source choices with explicit CI exit behavior. Inspect AI fits sandbox and agent labs. Hosted options include Braintrust and Confident AI. Pair any of them with tracing in Langfuse once traffic is live.

How is this different from evaluating LLM applications day to day?

The application playbook is how you discover failure modes and grow the suite. Evaluating LLMs before production is the release policy that turns that suite into a hard gate, plus rollback and online follow through. You need both.

Do I need ground truth labels before production?

You need enough trustworthy labels to support the decision. Production workflow labels are often noisy. GitHub advises treating them as signals and manually reviewing important or ambiguous subsets. Reference free checks (format, faithfulness style judges, trajectory rules) cover parts of the gate when labels are scarce, but high risk slices still need human review.