LLM as a judge vs agent as a judge comes down to what the grader can see and check. An LLM judge reads the text you hand it (an answer, or a pasted trace) and gives a verdict in one pass. An agent judge is given tools: it can open files, run code, search the web or query a database, then decide. Use an LLM judge when the evidence fits in the prompt, such as a chatbot reply or a summary. Switch to an agent judge when the truth lives outside the transcript, such as whether a coding agent's files actually work or whether a cited page really says what the answer claims. In the published tests, agent judges agreed with humans far more often on those tasks, but they cost more time and still make mistakes.

LLM as a judge vs agent as a judge at a glance

ItemLLM as a judgeAgent as a judge
What it readsThe output, optionally a trace you paste inThe output plus whatever it chooses to look up
ToolsNoneFile readers, code execution, search, browsers, database queries
Calls per verdictUsually oneSeveral to dozens
Best fitTone, helpfulness, faithfulness to a given context, rubric checks on textCoding agents, web research, GUI and data tasks, anything with side effects
Main failureTrusts what the text claimsMisreads tool output or reasons badly from good evidence
SpeedSecondsMinutes per task

Our LLM as a judge guide covers the basic method, including rubrics, bias and validation against human labels. Choosing which model sits in the judge seat is a separate question, answered in our LLM judge model guide. This page is about the step after that: whether your judge needs tools at all.

Where agent as a judge came from

The name comes from a paper by researchers at Meta AI and KAUST, Agent-as-a-Judge: Evaluate Agents with Agents, first posted in October 2024 and later presented at ICML 2025. Their problem was coding agents. Judging only the final result (did the task pass?) hides where an agent went wrong, and paying experts to read every workspace is slow.

So they built DevAI, a set of 55 realistic AI development tasks with 365 hierarchical requirements and 125 softer preferences. Three open source coding agents (MetaGPT, GPT-Pilot and OpenHands, all running on GPT-4o) attempted the tasks. Then three human experts, an LLM judge and an agent judge each checked which requirements were met.

The agent judge was modular: parts to map the project as a graph, locate the file a requirement refers to, read 33 file formats, search code, retrieve trajectory steps, plan and remember, plus a final "ask" step that made each yes or no call.

What the studies found

DevAI (2024). Against the human consensus, the agent judge agreed on about 90% of requirements and the LLM judge on about 70%. Judging OpenHands without access to its trajectory, the agent judge reached 90.44% agreement against 60.38% for the LLM judge. Individual human experts averaged between 84.88% and 89.34% agreement with their own group consensus, so the agent judge was in the range of a single careful human.

The ablation is the most useful part for builders. With only the ask step, the agent judge agreed 65.03% of the time. Adding the project graph lifted it to 75.95%, file reading to 82.24%, and locating the right file to 90.44%. Two additions made it worse: code search (down to 86.06%, likely from noisy retrieval) and memory of past judgments, because one early mistake carried into later verdicts. More tools are not automatically better.

The authors also warn that agreement rates can flatter a judge. MetaGPT rarely met any requirement, so an LLM judge that said "no" most of the time still scored 84.15% agreement. Look at precision and recall, not just percent agreement, when classes are lopsided.

AJ-Bench (2026). A team from the University of Science and Technology of China, the National University of Singapore and Meituan built AJ-Bench to test agent judges beyond code: 155 tasks and 516 labeled trajectories across web search, data systems (file systems and Postgres) and office GUIs, with 60 tools. Giving the same base model tools raised F1 by about 13 points on average. DeepSeek V3.2 with tools scored 77.34 F1, ahead of the best judge without tools, Gemini 3 Pro Preview at 75.05. The average agent judge still scored only about 0.72 F1, so the problem is far from solved.

Two findings run against intuition. Higher reasoning effort did not reliably help: GPT-5 mini did best at medium effort, and DeepSeek V3.2 was slightly worse in thinking mode. And when agent judges failed, the most common causes were misreading what a tool returned and reasoning badly from correct evidence, not forgetting to call a tool.

Mind2Web 2 (2025). Ohio State and Amazon researchers needed to grade long, citation heavy answers from deep research products, where the right answer changes over time. Mind2Web 2 uses a task specific judge agent for each of its 130 tasks. Each judge follows a rubric tree (50 nodes on average, up to 603) whose leaves are simple yes or no checks, such as "does this cited page support this price?" An LLM extracts facts and another call verifies them against cached pages. In a human audit of 720 checks, only 7 were real judge errors, a 99.03% correctness rate. The catch is the build cost: more than 1,000 hours of human work went into the tasks and the judge scripts.

A January 2026 survey on agent as a judge groups the methods into planning, tool based verification, several agents judging together, and persistent memory, and lists cost, latency, safety and privacy as the open problems.

What an agent judge costs

The DevAI paper gives the clearest numbers. Its three human experts spent 86.5 hours in total, about $1,297.50 at the $15 hourly rate the authors assumed. The agent judge cost $30.58 in API calls and took 118.43 minutes. The LLM judge cost almost the same, $29.63, because it was fed large contexts without the agent's file selection, but it finished in 10.99 minutes.

So in that study the agent judge was about ten times slower than the LLM judge for a similar bill, and far cheaper than people. Your numbers will differ with the number of tool calls per verdict. AJ-Bench found that a larger interaction budget kept improving F1, which means the most accurate settings are also the slowest. Its authors ran only two models as agent judges across the full benchmark because of budget, which tells you something about cost at scale.

There is also risk: a judge that runs code or browses needs the same sandbox as the agent it grades.

The middle option: an LLM judge that reads the trace

Most teams do not jump straight from grading final answers to a full agent judge. The common middle step is an LLM judge that reads the whole agent trace: every tool call, argument and result. The DevAI paper calls this the gray box setting, and it helped both judge types.

Common tools already do this:

  • DeepEval has a TaskCompletionMetric that uses an LLM judge on your agent's full trace to score whether the task was done. It needs DeepEval tracing on the agent (DeepEval docs).
  • LangChain's agentevals package offers create_trajectory_llm_as_judge with a built in trajectory accuracy prompt, plus trajectory match evaluators in strict, unordered, subset and superset modes (agentevals on GitHub). It plugs into LangSmith; our LangSmith with LangGraph guide shows the setup.
  • Langfuse and other tracing platforms store the traces these judges read. Our AI agent observability guide compares them.

We installed agentevals 0.0.9 and ran its four match modes on a toy refund agent that looked up the order but skipped the refund policy check before answering. Strict, unordered and superset all failed it, which is correct. Subset passed it, because subset only checks that the agent made no calls beyond the reference. That mode would miss a skipped step, so pick the mode that matches the failure you care about.

A trace reading judge still only sees what was logged. If the agent wrote a file, the trace says "file written," not whether the code inside runs. That gap is exactly where an agent judge earns its cost.

When to switch to an agent judge

Stay with an LLM judge (with the trace if you have one) when:

  • The thing you grade is text, and everything needed to grade it is in the prompt.
  • You need verdicts on thousands of outputs a day and seconds matter.
  • A deterministic check already covers correctness, and the judge only grades style or tone.

Move to an agent judge when:

  • The output is a workspace, a database change or a GUI state, and correctness means inspecting it.
  • Answers cite sources, and you need to confirm that each page supports each claim.
  • Answers change over time (prices, availability), so no fixed reference answer exists.
  • Your trace reading LLM judge keeps passing runs that humans later fail.

Before either, check whether a plain test would do. If a unit test, a SQL query or a file diff can decide pass or fail, run that and skip the model.

How to build a small agent judge

  1. Write the rubric as a tree of yes or no checks. DevAI and Mind2Web 2 both broke tasks into small binary requirements with dependencies. A failed critical check should fail its parent.
  2. Give the judge only the tools each check needs. Locating and reading files gave DevAI its biggest gains. Search and memory made it worse.
  3. Sandbox everything. Run the judge against a copy or replay of the final state, as AJ-Bench does for its file system and GUI tasks.
  4. Cap the tool calls per verdict. More turns helped AJ-Bench, but set a budget you can afford and log when it is hit.
  5. Log the judge's own trace. When a verdict looks wrong, you need to see whether it misread a tool result or reasoned badly, the two biggest failure types in AJ-Bench.
  6. Validate against human labels. Same rule as any LLM judge: score a labeled set and report precision and recall, not just agreement.

Inspect AI and similar frameworks already run agents in sandboxes, so they are a natural place to host a judge agent as well as the agent under test.

Limits of the evidence

Every number above comes from research teams grading their own benchmarks. DevAI's human baseline was three of the authors, and its cost math assumes a $15 hourly rate for experts. AJ-Bench was funded by Meituan and ran only two models as agent judges across all its tasks. Mind2Web 2's 99% figure applies to judges that humans refined by hand, task by task. None of these studies grades customer support or RAG agents in production. If your agent fails in ways like those in our Patronus AI agent failure guide, start with trace review and add tools to the judge only where the trace cannot answer the question. For tool calling skill itself, the public boards in our tool calling benchmark guide are a better reference.

FAQ

What is agent as a judge?

It is an evaluation method where an AI agent with tools grades the work of another AI system. Instead of reading only the final answer, it can open files, run code or check web pages before deciding. The term comes from a 2024 paper by Meta AI and KAUST researchers.

Is agent as a judge more accurate than LLM as a judge?

On the tasks tested so far, yes. DevAI reported about 90% agreement with human consensus against about 70% for an LLM judge, and AJ-Bench found about 13 points higher F1 for the same model with tools. AJ-Bench's average agent judge still scored only about 0.72 F1, so validate yours.

Is there an agent as a judge benchmark?

Yes. DevAI (55 coding tasks) was the first. AJ-Bench (155 tasks, 516 trajectories across search, data systems and GUIs) tests agent judges more broadly, and Mind2Web 2 uses judge agents to grade 130 web research tasks.

Can I use DeepEval or LangSmith for agent evaluation?

Yes, for trace based judging. DeepEval's Task Completion metric and LangChain's agentevals trajectory judge both use an LLM to read agent traces. A judge that calls tools of its own is usually custom code today.

Does agent as a judge cost more?

Usually in time, sometimes in money. In DevAI the agent judge cost $30.58 against $29.63 for the LLM judge, but took about 118 minutes instead of 11. Costs rise with the number of tool calls per verdict.