The Fireworks Specialized Intelligence Index (SII) is a free public set of leaderboards from Fireworks AI that scores language models on benchmarks built by companies that do the work themselves: Harvey for legal, Sierra for customer support, Grafana Labs for observability, Surge AI for tax and accounting, and dozens more. It does not produce one overall score. Each benchmark keeps its own metric, and models are ranked only within a benchmark or a domain. As of the October 28, 2026 update it covers 66 benchmarks across 14 domains, up from seven domains at launch on September 22.
The October 28 update is why the index is suddenly in the news. Surge AI added six benchmarks, including the new DAYJOB: Tax, whose launch post drew 257 points on Hacker News on October 28, and Replit added ViBench 2.0. This guide explains how the Fireworks Specialized Intelligence Index works, what its numbers mean, and where it can mislead you. If you want a single composite score instead, read our guide to the Artificial Analysis Intelligence Index v5, which takes the opposite approach.
What the Fireworks Specialized Intelligence Index measures
Fireworks says the index measures how models perform on domain benchmarks "under one standardized serving configuration and one harness per benchmark" (methodology). Two limits are written into that scope:
- It is not a test of top achievable performance. A team that tunes one model with a custom harness will usually beat the score shown.
- It is not a general intelligence index. Fireworks says plainly that it is "not a substitute for a generalist index."
The pitch in the launch post is that public benchmarks use bounded tasks and fixed datasets, while real work involves incomplete information and judgment calls. Fireworks cites METR's finding that maintainer acceptance of AI pull requests averaged 24.2 percentage points below automated SWE-bench Verified scores. It frames the gap as a chain: a benchmark score supports a capability claim only with more evidence, and a business outcome needs more evidence again. That is the same construct validity problem we cover in LLM benchmarks explained.
Fireworks does not design the benchmarks. Partners submit tasks and data in a Harbor compatible format, Fireworks reviews task diversity and difficulty, and the owner must attest to contamination controls. A benchmark is retired, or moved into a harder phase, once top models cluster together and it stops telling them apart.
The 14 domains and 66 benchmarks
Here is how the boards break down in the machine readable index as generated on October 28:
| Domain | Benchmarks | Examples |
|---|---|---|
| Software | 16 | SWE-Bench Pro V2, ViBench 2.0, sudo L7, o11y-bench, ORCA |
| Finance | 9 | DAYJOB: Tax, TaxCalcBench, Big Finance Benchmark |
| Legal | 7 | LAB (Harvey), Legora BAR, DAYJOB: Law |
| Customer Support | 5 | τ³-Banking, τ³-Voice, Cresta-Bench |
| Healthcare | 5 | Bedside Bench, HealthBench Professional |
| Web Search | 4 | ATLAS (Exa), Parallel Search |
| Security | 3 | dfbench, PWNBench v0.1, CyberGym |
| ML Research | 3 | FrontierMemory, WitnessBench |
| Life Sciences | 3 | BioBench, VIALS |
| Business Operations | 3 | AutomationBench (Zapier) |
| Marketing | 3 | AIM Routing, Grounding, Consistency |
| Productivity | 2 | Genspark Slides Benchmark |
| Hardware | 2 | CAD Arena |
| Chip Design | 1 | Build-A-Chip-Bench |
In words: software is by far the largest domain with 16 boards, finance has 9 and legal has 7. Forty five organizations are listed as contributors, with two more marked as coming soon. Forty four of the 66 boards show October 28 as their last update, so treat the index as a fast moving snapshot rather than a settled table.
How the index scores and ranks models
Native metrics, never averaged across benchmarks
Every board keeps its own headline metric. DAYJOB: Tax uses pass@1 at a 95% rubric threshold, o11y-bench uses pass^3 (the task must pass on all three attempts), Decagon's DuetBench uses an Elo rating, and Naïve's Clipping Bench reports a dollar figure. The methodology states that "metrics are never averaged across benchmarks" and that Fireworks does not build one overall score "as this would apply subjective weighting." Leaders are shown per domain, never across domains.
Domain ranks are average ranks
A model's domain rank is the plain mean of its ranks on the benchmarks in that domain, with every benchmark weighted equally. Missing results are not filled in or counted as zero. A model appears in a domain ranking only if it has scores on at least two of that domain's benchmarks, so one strong result cannot carry it to the top. Ties are broken by normalizing each result within its benchmark and averaging.
The run standard
Where Fireworks runs a benchmark itself, the default is three runs per task, reported as the average (avg@3) with a 95% confidence interval. pass@k (success on any of k tries) is never used. The default sampling profile, called SII-2026.09, sets temperature 1.0, top_p 0.95, a 131,072 token cap per call, maximum reasoning effort, and no fixed seed. Open weight models run on Fireworks serverless endpoints; closed models run through their developers' own APIs. Public benchmark repositories and known mirrors are blocked during runs, and every model must finish the same number of tasks.
Cost and duration sit beside the score. Cost is derived from token counts and a dated rate card; duration is wall clock time for the whole task, excluding judge calls. If you are new to these terms, our guide to how to evaluate LLM model performance explains pass rates and confidence intervals.
Who ran the numbers: the reproducibility tiers
This is the single most important column to read, and it is easy to miss. Each board carries one of three labels:
| Tier | Who can rerun it | Boards (Oct 28) |
|---|---|---|
| Public | Anyone: public data, a pinned open source harness, disclosed conditions | 15 |
| Fireworks | Fireworks and the owner: private data, but Fireworks ran a pinned harness | 8 |
| Partner | Only the owner, who supplied the score from its own evaluation | 43 |
So on 43 of 66 boards, about two thirds, the numbers come from the benchmark owner and Fireworks has not rerun them yet (the methodology says it plans to). Five of the six new Surge boards and Zapier's AutomationBench are owner supplied; Surge's DAYJOB: Healthcare is in the public tier. Replit's ViBench 2.0 is in the Fireworks tier, and Grafana's o11y-bench and Scale's SWE-Bench Pro V2 are in the public tier. A partner score can still be careful work, but it is a claim by the company that built the test, and you cannot check it from outside.
What changed on October 28
Six Surge AI benchmarks
Surge AI's announcement lists DAYJOB: Healthcare, Finance, Tax, Accounting and Data Science, plus sudo L7 for staff level software engineering. (The index also shows a Surge DAYJOB: Law board.) Surge says DAYJOB environments average 22 to 36 input files per task, that 44% of tasks are estimated to need 20 or more hours of professional work, and that roughly 70% of sudo L7 tasks come from privately licensed production codebases.
The DAYJOB: Tax post describes 50 environments across eight areas of tax practice and 1,789 files, including 1,487 PDFs, 117 Excel workbooks and a photo of a handwritten receipt. An attempt passes only when it meets at least 95% of an expert rubric, checked by an agent judge. The point is judgment: one task is a client asking which of two sale dates is better, where the real issue is a residence deadline the client never asked about.
The results are low. On the DAYJOB: Tax board, Claude Sonnet 5.5 leads at 10.8%, ahead of Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%, at about $10.71 per task for the leader. The best open model, DeepSeek V4.1 Flash (1017), scores 1.2%, and three of the 16 models score zero. A strict 95% threshold explains part of that: a nearly right return still fails. Compare it with Column Tax's TaxCalcBench on the same index, where the top model reaches 77.55% on exact line by line federal and state returns.
ViBench 2.0 from Replit
ViBench 2.0 starts the agent on an empty project and has it build a whole web app one feature at a time: 17 products, 241 feature requests and 544 test plans. A grading agent uses the finished app in a browser. On the index, the score is the share of builds where every test plan passes, averaged over four builds per app. GPT-6.1 Sol leads at 64.7% (about $12.34 per build), GPT-6 Astra follows at 60.3% (about $89.92) and Claude Opus 5.5 at 48.5%. All three open models tested (DeepSeek V4.1 Flash, GLM 5.3 and Kimi K3) score 0%. Eight apps are public in the ViBench repository and nine are held out.
An observability benchmark worth knowing: o11y-bench
For readers who run production systems, Grafana Labs' o11y-bench is the most practical board. Its 64 tasks run in a live Grafana stack with Prometheus, Loki and Tempo: querying metrics, logs and traces, investigating incidents, and building or repairing dashboards. Grading checks the agent's claims against facts read from the same environment, so a plausible answer alone fails.
Claude Opus 5.5 leads at 87.5% pass^3, but GPT-6.1 Sol sits at 85.9% for about $0.056 per task against $1.69, and GPT-6 Luna matches Astra at 84.4% for under a cent. Ten models land between 81% and 88%. That is a case where cost, not rank, should decide. Traversal's ORCA benchmark covers the harder root cause step: GPT-6 Astra leads its 324 task subset at 50.2%. If you are choosing tools for this work, see our AI agent observability guide.
How SII differs from Artificial Analysis and other leaderboards
| Item | Fireworks SII | Artificial Analysis Intelligence Index v5 |
|---|---|---|
| Overall score | None by design | One weighted composite |
| Who builds tests | Domain companies and public sources | A mix of its own and outside evals |
| Who runs them | Mixed: Fireworks, owners, or imported results | Artificial Analysis for every model |
| Metrics | Native per board | Combined into one weighted score |
| Best for | "Is this model good at my kind of work?" | "How do models compare in general?" |
The two answer different questions. A composite like Artificial Analysis is easier to quote but hides which tasks drove the number. SII keeps the detail but leaves the weighting to you. Our LLM leaderboards compared guide covers the general boards, and GDPval is the closest single benchmark to SII's real work idea.
SII also reveals saturation. Scale's SWE-Bench Pro V2 full public set shows its top three models within 0.3 points, led by Claude Opus 5 at 99.4%, so it barely separates frontier models any more. Newer boards like sudo L7, where Claude Sonnet 5.5 leads at 50.6%, still do.
How to read an SII board without fooling yourself
- Check the tier first. A partner score is the owner's number. Prefer Public or Fireworks boards when two disagree.
- Read the metric name. pass^3, pass@1 at 95% and Elo are not comparable, even inside one domain.
- Look at the model count. ViBench 2.0 tests nine models; MacroscopeBench tests 41. A first place among nine says less.
- Weigh cost and time. The table shows both, and on o11y-bench the gap between first and second is small while the price gap is about thirtyfold.
- Remember who publishes it. Fireworks sells inference and model training, open weight models are served on its own endpoints, and the index page promotes its training service. That does not make the scores wrong, but it is context.
- Note what the interval leaves out. Fireworks says no confidence interval covers judge variance, and many boards here use an agent or model judge. Our LLM as a judge guide explains why that matters.
FAQ
What is the Fireworks Specialized Intelligence Index?
It is a set of public leaderboards from Fireworks AI that test language models on benchmarks contributed by domain companies, such as Harvey (legal), Sierra (customer support) and Surge AI (tax, finance, healthcare). It launched on September 22, 2026 and covered 66 benchmarks in 14 domains as of October 28.
Does the SII give each model one overall score?
No. Fireworks deliberately avoids a cross domain composite because it would require subjective weights. Within a domain, a model's rank is the average of its ranks on that domain's benchmarks, and it must have results on at least two of them to be listed.
Who runs the SII benchmarks?
It depends on the board. Fireworks runs some with pinned open source harnesses, some results are imported from public leaderboards such as Scale's, and 43 of the 66 boards carry owner supplied scores that Fireworks has not yet rerun. Each board shows its reproducibility tier.
What is DAYJOB: Tax?
DAYJOB: Tax is a Surge AI benchmark of 50 professional tax assignments built from realistic client requests and records, released on October 27, 2026. An attempt passes only if it meets 95% of an expert rubric. The top model on the SII board, Claude Sonnet 5.5, passes 10.8% of the time.
Is the Specialized Intelligence Index free to use?
Yes. The leaderboards are free to browse, and Fireworks publishes the full results as JSON and Markdown files. Benchmark owners can submit tasks in a Harbor compatible format, and Fireworks says it runs the evaluation across its model roster at no cost.
How is SII different from the Artificial Analysis Intelligence Index?
Artificial Analysis runs a fixed set of evaluations on every model and combines them into one weighted score. SII collects many domain benchmarks, keeps each one's native metric and leaves the weighting to the reader. Use the first for a general picture and the second to check a specific kind of work.
