Patronus AI agent failure research boils down to two findings. First, small errors compound: if an agent has a 1% chance of a mistake at each step, the chance that something goes wrong somewhere in a 100 step run is about 63%. Second, models are poor at finding those mistakes after the fact. On Patronus AI's TRAIL benchmark of 148 hand labeled agent traces, the best model tested located and classified errors correctly only about 11% of the time. The 63% number is arithmetic from an assumed error rate, not a measured failure rate, and Patronus has since retired its Percival debugger to focus on training environments.
This guide separates the headline from the research, walks through the TRAIL error taxonomy, and turns it into a checklist for debugging your own agents. Everything was checked against Patronus AI's papers, docs and announcements on November 7, 2026. For the wider tool picture, see our guide to AI agent observability.
Patronus AI agent failure research at a glance
| Item | What it is | Status in November 2026 |
|---|---|---|
| 63% compounding figure | Patronus framing: 1% error per step compounds to a 63% chance of error by step 100 | Arithmetic illustration, widely quoted |
| TRAIL benchmark | 148 annotated traces, 841 labeled errors, an error taxonomy | Open (MIT), gated dataset on Hugging Face |
| Percival | Agent that read traces and flagged 20+ failure modes | Retired, per Patronus docs |
| Generative Simulators | Adaptive training environments for agents (December 2025) | Folded into the environment work |
| Digital World Models | Simulated software worlds for agent training (June 2026) | Current focus, with a $50M Series B |
Patronus AI was founded in 2023 by former Meta AI researchers Anand Kannappan and Rebecca Qian, according to TechCrunch. It began as an evaluation company. Its homepage now calls it "a behavioral research lab for AI" that studies where models fail and how to improve them.
Where the "AI agents fail 63% of the time" headline comes from
In December 2025, VentureBeat ran the headline "AI agents fail 63% of the time on complex tasks" over a story about Patronus AI's Generative Simulators. The body of the article is more careful. It says research published earlier that year found that an agent with a 1% error rate per step can compound to a 63% chance of failure by the hundredth step. The TRAIL paper traces the idea to a 2025 Patronus blog post on modeling statistical risk in AI products.
The math is simple. If each step succeeds 99% of the time and the steps are independent, the chance of 100 clean steps is 0.99 to the power of 100, which is about 36.6%. So the chance of at least one error is about 63.4%. We ran the same formula for a few other cases:
| Error chance per step | Steps | Chance of at least one error |
|---|---|---|
| 1% | 10 | about 9.6% |
| 0.5% | 100 | about 39.4% |
| 1% | 100 | about 63.4% |
| 2% | 50 | about 63.6% |
Three caveats matter when you see the 63% figure quoted:
- It is a model, not a measurement. Nobody observed agents failing 63% of the time. The number follows from an assumed 1% per step error rate.
- An error is not always a failed task. Good agents notice a bad tool result and retry. TRAIL's annotators rated about 44% of output generation errors as low impact.
- Errors are not independent. One wrong assumption early can poison every later step, which makes things worse than the formula, while retries and checks can make them better.
The useful lesson survives the caveats: per step reliability has to be very high before long agent runs become dependable, and you need a way to see which step went wrong.
What the TRAIL benchmark measured
TRAIL (Trace Reasoning and Agentic Issue Localization) is a May 2025 paper by Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan and Rebecca Qian. It asks a narrow question: given the full trace of an agent run, can a model find where it went wrong and name the type of error?
How the dataset was built, from the paper and the Patronus launch post:
- 148 traces: 117 from GAIA (open web research) and 31 from SWE-bench Lite (fixing GitHub issues), per the Hugging Face dataset card.
- Real agent stacks. GAIA traces came from Hugging Face's open deep research agent with
o3-minias the manager and search agents. SWE-bench traces came from a CodeAct agent running on Claude 3.7 Sonnet. - OpenTelemetry format. Traces were captured with the OpenInference standard, so they look like what an observability tool would store: 1,987 spans in total, 575 of them with at least one error.
- 841 labeled errors, an average of 5.68 per trace. Annotation plus review took about 110 minutes per trace.
- Very long inputs. Patronus says the average trace is over 200,000 input tokens and the longest runs into the millions, beyond the context window of several models.
The task for the model under test is to output a JSON list of errors, each with a category from the taxonomy, the span ID where it happened, evidence and an impact rating. Scores cover category F1, location accuracy and joint accuracy (right category at the right span).
The TRAIL agent error taxonomy
TRAIL sorts agent errors into three groups and about 20 leaf types. This is the most reusable part of the research, because you can label your own traces with it.
| Group | Categories | Example leaf errors |
|---|---|---|
| Reasoning | Hallucinations, information processing, decision making, output generation | Fabricated tool output, tool output misread, wrong tool chosen, formatting errors, instruction non compliance |
| System execution | Configuration, API issues, resource management | Bad tool definition, missing API key, 429 rate limit, 401 or 403 auth error, 500 service error, 404, timeouts |
| Planning and coordination | Context management, task management | Context handling failures, repeated tool calls, goal deviation, task orchestration between agents |
The prompt used in the paper also allows a fourth bucket for domain specific errors. Two findings from the dataset stand out. Formatting errors and instruction non compliance made up 353 of the 841 errors, nearly 42%. System execution errors were rare, but the authors flag them as the ones agents struggle to recover from.
How well models found agent failures
The results table in the paper is where the 11% comes from. Joint accuracy means the model named the right error category at the right span.
| Model (reasoning set to high where available) | Joint accuracy, GAIA split | Joint accuracy, SWE-bench split |
|---|---|---|
| Gemini 2.5 Pro Preview | 18.3% | 5.0% |
| Gemini 2.5 Flash Preview | 10.0% | 0% |
| OpenAI o3 | 9.2% | context too long |
| Claude 3.7 Sonnet | 4.7% | context too long |
| GPT-4.1 | 2.8% | 0% |
Averaged across both splits, Gemini 2.5 Pro reached about 11%. Other details from the paper are just as telling:
- Performance fell as traces got longer. All three metrics were negatively correlated with input length.
- More reasoning helped. For o3, category F1 dropped from 0.296 to 0.264 as reasoning effort went from high to low.
- Context handling failures were almost never caught. Nearly every model scored an F1 of 0.00 on them, with Claude 3.7 Sonnet the exception at 0.18.
- Three of the eight models could not fit the SWE-bench traces in context at all.
These are mid 2025 models, and newer ones will score higher. The structural point holds anyway: asking one model to read a whole agent trace and judge it is the weakest way to find failures. That matches what we found on LLM as a judge reliability more broadly.
Percival, Generative Simulators and Digital World Models
Patronus first tried to productize this research. VentureBeat reported the launch of Percival, an agent that read traces, detected more than 20 failure modes across reasoning, system execution, planning and domain specific categories, and suggested fixes. Patronus said early customers cut trace review from about an hour to one to 1.5 minutes.
That product is gone. The Patronus docs now say the LLM evaluation platform "is retired and no longer offered" and that Percival is retired, pointing readers to TRAIL instead. If you find older guides recommending Patronus as a hosted judge or monitoring service, treat that as out of date.
The company moved upstream into training:
- Generative Simulators (December 2025). Adaptive environments that generate new tasks and adjust difficulty as an agent learns. Patronus told VentureBeat that training on its environments raised task completion by 10% to 20% on software engineering, customer service and financial analysis tasks. That is a company claim without a public benchmark behind it.
- Digital World Models (June 25, 2026). Patronus announced a $50M Series B led by Greenfield Partners, bringing total funding to $70 million, and previewed language diffusion world models that simulate software, research and communication workflows for agents to practice in.
- Post training results (October 29, 2026). In a post training writeup, Patronus trained Qwen3.8-27B on its tasks. Held out GitHub task success rose from 35.6% to 47.8%, which fixed about a fifth of the starting model's failures. The held out sets were small (46 GitHub and 54 Unreal Engine tasks).
The October post also includes failure lessons any agent team can use. An environment outage made every attempt fail silently with zero reward instead of crashing the job, and a grader passed two runs that had actually hit the turn limit before finishing.
Other agent failure research worth knowing
TRAIL is not the only taxonomy. UC Berkeley's MAST study ("Why Do Multi Agent LLM Systems Fail?") annotated 1,642 traces from seven multi agent frameworks and found failure rates of 41% to 86.7% on its tasks. It groups 14 failure modes into system design issues, inter agent misalignment and task verification, and it ships an LLM annotator as the agentdash Python package.
MAST is stronger on coordination failures between agents, such as withholding information or ignoring another agent's input. TRAIL is stronger on system errors such as rate limits and bad tool definitions. Used together they cover most of what goes wrong. If your failures are mainly about calling tools correctly, our guide to which LLM benchmark measures tool calling covers BFCL and Tau bench.
How to find failures in your own agents
The research points to a practical workflow that does not depend on any one vendor:
- Trace every step. Capture LLM calls, tool calls and their results as OpenTelemetry spans. Langfuse, Arize Phoenix and LangSmith all store agent traces; Phoenix maintains OpenInference, the format TRAIL used.
- Catch system errors with code, not judges. Status codes like 429, 401, 403, 404 and 500, timeouts and malformed JSON are cheap to detect with rules. TRAIL shows these are rare but costly.
- Count steps and repeats. Flag runs with unusually many steps or the same tool called again and again. Repeated calls map to TRAIL's resource abuse category and MAST's step repetition.
- Hand label a sample. Take 20 to 50 failed runs and tag each with a TRAIL or MAST category and the span where it started. This tells you which failure type dominates for your agent.
- Judge spans, not whole traces. If you use a model to grade, give it one step plus the context it needs, not a 200,000 token trace. Eval frameworks such as DeepEval and Inspect AI let you write custom scorers that run on a single step.
- Check your grader. As Patronus found, a pass can hide an unfinished run. Spot check passing traces too.
For the evaluation side of this loop, our LLM hallucination detection guide covers checks for fabricated tool output and ungrounded answers.
FAQ
Do AI agents really fail 63% of the time?
No study measured that. The 63% figure is the chance of at least one error in 100 steps if each step has an independent 1% error rate. Real failure rates depend on the task, the model and whether the agent can recover from mistakes.
What is the Patronus AI TRAIL benchmark?
TRAIL is a 2025 benchmark of 148 agent traces from GAIA and SWE-bench Lite with 841 hand labeled errors. It tests whether a model can find the span where an agent went wrong and name the error type. The best model tested scored about 11% joint accuracy.
Is Patronus Percival still available?
No. Patronus AI's docs say its LLM evaluation platform and Percival are retired and no longer offered. The company now builds training data and simulated environments for agents.
Can I download the TRAIL dataset?
Yes. The code is MIT licensed on GitHub, and the dataset is on Hugging Face behind a click through gate. You agree not to reshare it outside a gated or private repository, to limit contamination.
What are the most common agent failures in TRAIL?
Output generation errors dominate. Formatting errors and instruction non compliance account for nearly 42% of the 841 labeled errors. API and configuration errors are rarer but harder for an agent to recover from.
How is TRAIL different from the MAST taxonomy?
MAST, from UC Berkeley, focuses on multi agent systems and has 14 failure modes grouped into system design, inter agent misalignment and task verification. TRAIL covers single and multi agent traces and adds detailed system execution errors such as rate limits and authentication failures.


