The LLM benchmark that measures tool calling most directly is the Berkeley Function Calling Leaderboard (BFCL), now in version 4. BFCL scores whether a model picks the right function, fills arguments correctly, handles parallel and multi turn calls, abstains when no tool fits, and (in V4) uses web search and memory tools in agentic setups. For tool use inside a simulated customer conversation, pair it with Sierra's Tau bench family. For end to end shell jobs rather than structured API calls, use Terminal-Bench instead.
This guide answers which board to cite for tool calling, how BFCL V4 is scored, how it differs from ToolBench, Tau bench and coding agent boards, and how to read a published Overall Acc number without mixing modes or categories. It is a methodology explainer, not a model buying guide. For the wider map of knowledge and coding suites, start with LLM benchmarks explained. For how specialist boards sit next to Arena style preference rankings, see LLM leaderboards compared.
What "tool calling" means on a benchmark
Tool calling (also called function calling) is the skill of emitting a structured call to an external function or API: the right name, the right arguments, and the right decision about whether to call anything at all. It is not the same as answering a multiple choice exam, writing free form chat text or finishing a long coding job in a terminal.
A useful tool calling score answers questions like:
- Did the model select the correct function from the inventory?
- Were the argument names and types valid against the schema?
- Could it issue several independent calls in one turn when needed?
- Could it continue across turns when a result came back incomplete?
- Did it refuse to call a tool when none of the options fit (irrelevance)?
Agentic boards stretch those questions further: can the model search the web, read or write memory, and keep going when a fetch fails? BFCL V4 is built for that stack. Coding agent boards such as SWE-Bench Pro and Terminal-Bench also use tools, but they score the final patch or environment state, not the correctness of each structured call.
The short answer: BFCL is the public standard
When a lab or tracker says a model is strong at tool use or function calling, the board they almost always mean is BFCL. UC Berkeley's Gorilla project hosts the live leaderboard, publishes the dataset on Hugging Face, and ships a reproducible eval package (bfcl-eval). The ICML 2025 paper, The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (Patil et al.), is the archival citation.
BFCL grew in four public generations:
| Version | What it added | Scoring idea |
|---|---|---|
| V1 | Simple, multiple, parallel and parallel multiple single turn calls | AST match on the call tree |
| V2 | Live enterprise and open source contributed functions | Same AST idea on fresher inventories |
| V3 | Multi turn and multi step dialogues (missing args, missing functions, long context) | State aware multi turn checks |
| V4 | Agentic web search and memory, plus format sensitivity reporting | Weighted Overall Acc across categories |
So if your question is literally "which LLM benchmark measures tool calling?", cite BFCL V4 and link the official board. Use other suites when your product question is different (dialogue policy, huge API catalogs, or shell completion).
How BFCL V4 scores Overall Acc
The V4 web search blog (July 2025, updated July 20, 2025) publishes the composition of the headline Overall Score. Main categories use fixed percentage weights:
- Agentic: 40% (web search and memory averaged equally)
- Multi Turn: 30% (base, missing function, missing parameter, long context averaged equally)
- Live: 10% (live AST categories weighted by entry count)
- Non Live: 10% (Python, Java and JavaScript simple AST plus multiple, parallel and parallel multiple)
- Hallucination measurement: 10% (irrelevance detection averaged across live and non live)
Format sensitivity (dozens of prompt and schema format variants) is reported on the board but does not enter the Overall Score. Live relevance is also shown separately and is not weighed into the headline number.
AST (Abstract Syntax Tree) matching is the core single turn idea: the eval parses the model's call into a tree and checks names and arguments without always executing the real API. Live categories add community contributed functions. Multi turn categories check that the conversation and tool state move correctly when parameters or functions are missing. Web search gives the model DuckDuckGo search plus a URL fetch tool on 100 human crafted multihop questions. Memory tests read and write across vector, key value and recursive summary stores.
Read category columns, not only the overall percentage. Simple single turn AST is close to saturated for frontier models. Multi turn, web search and memory are where gaps still show up. A model can look strong on Non Live AST and weak once the dialogue needs clarification or a search hop.
FC mode vs Prompt mode
BFCL labels each row as FC or Prompt:
- FC: the vendor's native function or tool calling API.
- Prompt: the model is asked to emit calls through ordinary text generation (a workaround when native tool calling is weak or unavailable).
Those are different experiments. On the overall CSV we checked on November 6, 2026, Claude Opus 4.5 in FC mode led at 77.47% Overall Acc, while the same model family in Prompt mode sat far lower on the same board. Always quote the mode with the score. Prefer the FC row when you care about production tool APIs; use Prompt rows when you need a fair comparison for models without a mature tool interface.
Snapshot of the public board (November 6, 2026)
Top Overall Acc rows from Berkeley's published CSV on November 6, 2026 (FC preferred when a model appears twice):
| Rank | Model (mode) | Overall Acc |
|---|---|---|
| 1 | Claude Opus 4.5 (FC) | 77.47% |
| 2 | Claude Sonnet 4.5 (FC) | 73.24% |
| 3 | Gemini 3 Pro Preview (Prompt) | 72.51% |
| 4 | GLM 4.6 (FC thinking) | 72.38% |
| 5 | Grok 4.1 Fast Reasoning (FC) | 69.57% |
Source: gorilla.cs.berkeley.edu/data_overall.csv. Rankings move as labs submit new runs. Treat a few points as a soft gap, and always open the category columns before you shortlist a model for an agent product.
Cost and latency columns on the same CSV estimate the dollars and seconds for a full benchmark pass. They are useful for rough comparison, not for your production unit economics.
Related benchmarks that are not BFCL
Several other suites appear in tool use papers. None replace BFCL as the default "does this model call tools correctly?" board, but each answers a different question.
Tau bench and later Tau versions
Sierra's Tau bench (Tool Agent User Interaction Benchmark) puts an agent in a customer service domain with a policy document, domain APIs and a simulated user. Success is whether the final database state matches the annotated goal, not whether each call matched an AST. The original ICLR 2025 paper reported that even strong function calling agents succeeded on under half the tasks and were inconsistent across trials (pass^k). Successors (Tau2 / Tau3) add domains, knowledge retrieval and voice. Use Tau bench when your product is a policy bound dialogue agent. Use BFCL when you need a clean score for structured call accuracy.
ToolBench and ToolEval
ToolBench (Qin et al., ToolLLM / ICLR 2024) is a large scale training and eval set built from more than 16,000 real RapidAPI endpoints. ToolEval scores pass rate and win rate with an LLM judge. It remains a foundational reference for breadth across huge API inventories. For a reproducible public leaderboard of call correctness in 2026, BFCL is the clearer citation.
API Bank
API Bank splits tool use into planning, retrieval and calling across a smaller hand annotated set. It is useful in research papers that want those three skills separated. It is not the board most trackers quote for a headline tool calling number.
Terminal-Bench, SWE-Bench Pro and Aider
Terminal-Bench scores whether an agent finishes a sandboxed shell job. SWE-Bench Pro scores whether a repository patch passes hidden tests. The Aider leaderboards score real file edits in a coding harness. All of them use tools. None of them are a pure measure of structured function call accuracy the way BFCL is. If your product is a coding agent, those boards matter more than BFCL for the final outcome; still run BFCL when you need to know whether the model can speak your tool schema at all.
When2Call
NVIDIA's When2Call tests the decision before the call: given a conversation and a set of tools, should the assistant answer directly, call a tool, ask for more information or say it cannot answer? Artificial Analysis runs a filtered set of 3,423 of its multiple choice tasks in its Decision Index; in v2, released November 8, 2026, it is one of six evaluations, at 10% weight. It complements BFCL, which checks whether the call itself is well formed.
Composite indexes
Artificial Analysis and LiveBench mix many evals into capability indexes. They may include tool related tasks, but a single index point is not a substitute for a BFCL Overall Acc when the question is specifically tool calling. OpenCompass can run many evals in one harness; check which tool suite you enabled before you quote a number.
How to choose the right board for your question
| Your question | Prefer |
|---|---|
| Does the model emit correct function calls? | BFCL V4 Overall Acc, plus multi turn and agentic columns |
| Can it follow a domain policy while talking to a user? | Tau bench / Tau3 |
| Can it finish a shell or repo coding job? | Terminal-Bench, SWE-Bench Pro, Aider |
| How does it rank on a mixed capability index? | Artificial Analysis, LiveBench (then dig into the tool slice) |
| Will it work on our tools? | Your own schema tests in CI (see how to evaluate LLM model performance) |
Public boards shortlist models. Your inventory, argument types, auth errors and latency budget decide. A model that leads BFCL can still fail if your tools return messy errors, need strict enum values or sit behind custom wrappers.
How to read a BFCL claim without getting fooled
- Name the version. V3 and V4 are not comparable as one number.
- Name FC or Prompt. Never average them.
- Open the category table. Multi turn and agentic columns matter more than saturated simple AST.
- Check the commit or package pin. The leaderboard page records the eval commit and
bfcl-evalversion used for published rows. - Separate cost and latency from accuracy. Cheap and fast rows can be weak on multi turn.
- Do not treat Format Sensitivity as part of Overall Acc. It is diagnostic.
- Retest on your schemas. BFCL functions are not your product APIs.
For a bake off workflow that adds error bars and prompt sensitivity on your own set, follow how to evaluate LLM model performance. For a release gate before production traffic, see how to evaluate LLMs before production.
Limits of tool calling benchmarks
BFCL is excellent at structured call correctness and increasingly strong at agentic search and memory. It still does not measure your brand voice, your safety policy, your retrieval quality or whether a human would trust the final answer. Tau bench adds user interaction and policy, but only inside a few domains. ToolBench's scale is impressive, yet LLM judged win rates are noisier than AST matches. Coding agent boards score outcomes, so a wrong intermediate call that still reaches a green test can hide tool bugs.
Contamination and overfitting apply here too, as they do for every public suite (see the contamination section in LLM benchmarks explained). Prefer independent rows over vendor cherry picks, and keep a private holdout of your own tool traces.
FAQ
Which LLM benchmark measures tool calling?
The Berkeley Function Calling Leaderboard (BFCL), currently V4. It is the public standard for scoring whether models select functions and fill arguments correctly, including multi turn and agentic web search and memory categories.
Is BFCL the same as Tau bench?
No. BFCL scores structured function call correctness (plus V4 agentic tools). Tau bench scores whether a dialogue agent reaches the right database state while following a domain policy with a simulated user. Many teams run both.
What does BFCL Overall Acc include in V4?
A weighted mix: Agentic 40%, Multi Turn 30%, Live 10%, Non Live 10%, Hallucination (irrelevance) 10%. Format sensitivity is reported separately and does not change Overall Acc, per the official V4 scoring writeup.
Should I quote FC or Prompt scores?
Quote the mode that matches how you will call the model. For production tool APIs, prefer FC. Only compare Prompt rows to other Prompt rows.
Does Terminal-Bench measure tool calling?
Not as a pure function calling score. Terminal-Bench measures whether an agent finishes sandboxed shell tasks. Agents use tools there, but the pass signal is the final environment state, not AST matched API calls. Use BFCL when you need the latter.
How do I test tool calling on my own APIs?
Build a fixed set of prompts with expected call traces for your schemas, run two or more models with identical settings, score argument validity and task success, and put the suite in CI. Public BFCL scores shortlist candidates; your schemas decide. Our model performance and pre production guides walk through that pattern.


