Patronus AI agent failure research boils down to two findings. First, small errors compound: if an agent has a 1% chance of a mistake at each step, the chance that something goes wrong somewhere in a 100 step run is about 63%. Second, models are poor at finding those mistakes after the fact. On Patronus AI's TRAIL benchmark of 148 hand labeled agent traces, the best model tested located and classified errors correctly only about 11% of the time. The 63% number is arithmetic from an assumed error rate, not a measured failure rate, and Patronus has since retired its Percival debugger to focus on training environments.

This guide separates the headline from the research, walks through the TRAIL error taxonomy, and turns it into a checklist for debugging your own agents. Everything was checked against Patronus AI's papers, docs and announcements on November 7, 2026. For the wider tool picture, see our guide to AI agent observability.

Patronus AI agent failure research at a glance

ItemWhat it isStatus in November 2026
63% compounding figurePatronus framing: 1% error per step compounds to a 63% chance of error by step 100Arithmetic illustration, widely quoted
TRAIL benchmark148 annotated traces, 841 labeled errors, an error taxonomyOpen (MIT), gated dataset on Hugging Face
PercivalAgent that read traces and flagged 20+ failure modesRetired, per Patronus docs
Generative SimulatorsAdaptive training environments for agents (December 2025)Folded into the environment work
Digital World ModelsSimulated software worlds for agent training (June 2026)Current focus, with a $50M Series B

Patronus AI was founded in 2023 by former Meta AI researchers Anand Kannappan and Rebecca Qian, according to TechCrunch. It began as an evaluation company. Its homepage now calls it "a behavioral research lab for AI" that studies where models fail and how to improve them.

Where the "AI agents fail 63% of the time" headline comes from

In December 2025, VentureBeat ran the headline "AI agents fail 63% of the time on complex tasks" over a story about Patronus AI's Generative Simulators. The body of the article is more careful. It says research published earlier that year found that an agent with a 1% error rate per step can compound to a 63% chance of failure by the hundredth step. The TRAIL paper traces the idea to a 2025 Patronus blog post on modeling statistical risk in AI products.

The math is simple. If each step succeeds 99% of the time and the steps are independent, the chance of 100 clean steps is 0.99 to the power of 100, which is about 36.6%. So the chance of at least one error is about 63.4%. We ran the same formula for a few other cases:

Error chance per stepStepsChance of at least one error
1%10about 9.6%
0.5%100about 39.4%
1%100about 63.4%
2%50about 63.6%

Three caveats matter when you see the 63% figure quoted:

  • It is a model, not a measurement. Nobody observed agents failing 63% of the time. The number follows from an assumed 1% per step error rate.
  • An error is not always a failed task. Good agents notice a bad tool result and retry. TRAIL's annotators rated about 44% of output generation errors as low impact.
  • Errors are not independent. One wrong assumption early can poison every later step, which makes things worse than the formula, while retries and checks can make them better.

The useful lesson survives the caveats: per step reliability has to be very high before long agent runs become dependable, and you need a way to see which step went wrong.

What the TRAIL benchmark measured

TRAIL (Trace Reasoning and Agentic Issue Localization) is a May 2025 paper by Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan and Rebecca Qian. It asks a narrow question: given the full trace of an agent run, can a model find where it went wrong and name the type of error?

How the dataset was built, from the paper and the Patronus launch post:

  • 148 traces: 117 from GAIA (open web research) and 31 from SWE-bench Lite (fixing GitHub issues), per the Hugging Face dataset card.
  • Real agent stacks. GAIA traces came from Hugging Face's open deep research agent with o3-mini as the manager and search agents. SWE-bench traces came from a CodeAct agent running on Claude 3.7 Sonnet.
  • OpenTelemetry format. Traces were captured with the OpenInference standard, so they look like what an observability tool would store: 1,987 spans in total, 575 of them with at least one error.
  • 841 labeled errors, an average of 5.68 per trace. Annotation plus review took about 110 minutes per trace.
  • Very long inputs. Patronus says the average trace is over 200,000 input tokens and the longest runs into the millions, beyond the context window of several models.

The task for the model under test is to output a JSON list of errors, each with a category from the taxonomy, the span ID where it happened, evidence and an impact rating. Scores cover category F1, location accuracy and joint accuracy (right category at the right span).

The TRAIL agent error taxonomy

TRAIL sorts agent errors into three groups and about 20 leaf types. This is the most reusable part of the research, because you can label your own traces with it.

GroupCategoriesExample leaf errors
ReasoningHallucinations, information processing, decision making, output generationFabricated tool output, tool output misread, wrong tool chosen, formatting errors, instruction non compliance
System executionConfiguration, API issues, resource managementBad tool definition, missing API key, 429 rate limit, 401 or 403 auth error, 500 service error, 404, timeouts
Planning and coordinationContext management, task managementContext handling failures, repeated tool calls, goal deviation, task orchestration between agents

The prompt used in the paper also allows a fourth bucket for domain specific errors. Two findings from the dataset stand out. Formatting errors and instruction non compliance made up 353 of the 841 errors, nearly 42%. System execution errors were rare, but the authors flag them as the ones agents struggle to recover from.

How well models found agent failures

The results table in the paper is where the 11% comes from. Joint accuracy means the model named the right error category at the right span.

Model (reasoning set to high where available)Joint accuracy, GAIA splitJoint accuracy, SWE-bench split
Gemini 2.5 Pro Preview18.3%5.0%
Gemini 2.5 Flash Preview10.0%0%
OpenAI o39.2%context too long
Claude 3.7 Sonnet4.7%context too long
GPT-4.12.8%0%

Averaged across both splits, Gemini 2.5 Pro reached about 11%. Other details from the paper are just as telling:

  • Performance fell as traces got longer. All three metrics were negatively correlated with input length.
  • More reasoning helped. For o3, category F1 dropped from 0.296 to 0.264 as reasoning effort went from high to low.
  • Context handling failures were almost never caught. Nearly every model scored an F1 of 0.00 on them, with Claude 3.7 Sonnet the exception at 0.18.
  • Three of the eight models could not fit the SWE-bench traces in context at all.

These are mid 2025 models, and newer ones will score higher. The structural point holds anyway: asking one model to read a whole agent trace and judge it is the weakest way to find failures. That matches what we found on LLM as a judge reliability more broadly.

Percival, Generative Simulators and Digital World Models

Patronus first tried to productize this research. VentureBeat reported the launch of Percival, an agent that read traces, detected more than 20 failure modes across reasoning, system execution, planning and domain specific categories, and suggested fixes. Patronus said early customers cut trace review from about an hour to one to 1.5 minutes.

That product is gone. The Patronus docs now say the LLM evaluation platform "is retired and no longer offered" and that Percival is retired, pointing readers to TRAIL instead. If you find older guides recommending Patronus as a hosted judge or monitoring service, treat that as out of date.

The company moved upstream into training:

  • Generative Simulators (December 2025). Adaptive environments that generate new tasks and adjust difficulty as an agent learns. Patronus told VentureBeat that training on its environments raised task completion by 10% to 20% on software engineering, customer service and financial analysis tasks. That is a company claim without a public benchmark behind it.
  • Digital World Models (June 25, 2026). Patronus announced a $50M Series B led by Greenfield Partners, bringing total funding to $70 million, and previewed language diffusion world models that simulate software, research and communication workflows for agents to practice in.
  • Post training results (October 29, 2026). In a post training writeup, Patronus trained Qwen3.8-27B on its tasks. Held out GitHub task success rose from 35.6% to 47.8%, which fixed about a fifth of the starting model's failures. The held out sets were small (46 GitHub and 54 Unreal Engine tasks).

The October post also includes failure lessons any agent team can use. An environment outage made every attempt fail silently with zero reward instead of crashing the job, and a grader passed two runs that had actually hit the turn limit before finishing.

Other agent failure research worth knowing

TRAIL is not the only taxonomy. UC Berkeley's MAST study ("Why Do Multi Agent LLM Systems Fail?") annotated 1,642 traces from seven multi agent frameworks and found failure rates of 41% to 86.7% on its tasks. It groups 14 failure modes into system design issues, inter agent misalignment and task verification, and it ships an LLM annotator as the agentdash Python package.

MAST is stronger on coordination failures between agents, such as withholding information or ignoring another agent's input. TRAIL is stronger on system errors such as rate limits and bad tool definitions. Used together they cover most of what goes wrong. If your failures are mainly about calling tools correctly, our guide to which LLM benchmark measures tool calling covers BFCL and Tau bench.

How to find failures in your own agents

The research points to a practical workflow that does not depend on any one vendor:

  1. Trace every step. Capture LLM calls, tool calls and their results as OpenTelemetry spans. Langfuse, Arize Phoenix and LangSmith all store agent traces; Phoenix maintains OpenInference, the format TRAIL used.
  2. Catch system errors with code, not judges. Status codes like 429, 401, 403, 404 and 500, timeouts and malformed JSON are cheap to detect with rules. TRAIL shows these are rare but costly.
  3. Count steps and repeats. Flag runs with unusually many steps or the same tool called again and again. Repeated calls map to TRAIL's resource abuse category and MAST's step repetition.
  4. Hand label a sample. Take 20 to 50 failed runs and tag each with a TRAIL or MAST category and the span where it started. This tells you which failure type dominates for your agent.
  5. Judge spans, not whole traces. If you use a model to grade, give it one step plus the context it needs, not a 200,000 token trace. Eval frameworks such as DeepEval and Inspect AI let you write custom scorers that run on a single step.
  6. Check your grader. As Patronus found, a pass can hide an unfinished run. Spot check passing traces too.

For the evaluation side of this loop, our LLM hallucination detection guide covers checks for fabricated tool output and ungrounded answers.

FAQ

Do AI agents really fail 63% of the time?

No study measured that. The 63% figure is the chance of at least one error in 100 steps if each step has an independent 1% error rate. Real failure rates depend on the task, the model and whether the agent can recover from mistakes.

What is the Patronus AI TRAIL benchmark?

TRAIL is a 2025 benchmark of 148 agent traces from GAIA and SWE-bench Lite with 841 hand labeled errors. It tests whether a model can find the span where an agent went wrong and name the error type. The best model tested scored about 11% joint accuracy.

Is Patronus Percival still available?

No. Patronus AI's docs say its LLM evaluation platform and Percival are retired and no longer offered. The company now builds training data and simulated environments for agents.

Can I download the TRAIL dataset?

Yes. The code is MIT licensed on GitHub, and the dataset is on Hugging Face behind a click through gate. You agree not to reshare it outside a gated or private repository, to limit contamination.

What are the most common agent failures in TRAIL?

Output generation errors dominate. Formatting errors and instruction non compliance account for nearly 42% of the 841 labeled errors. API and configuration errors are rarer but harder for an agent to recover from.

How is TRAIL different from the MAST taxonomy?

MAST, from UC Berkeley, focuses on multi agent systems and has 14 failure modes grouped into system design, inter agent misalignment and task verification. TRAIL covers single and multi agent traces and adds detailed system execution errors such as rate limits and authentication failures.