VIALS-AA is Artificial Analysis's independent run of Handshake AI's VIALS benchmark, a 161 task test of whether models can interpret scientific images from life sciences work (gel blots, plasmid maps, flow cytometry plots and related artifacts) and answer a short question about the evidence they contain. Artificial Analysis launched the public board on November 6, 2026. It runs every task three times in its open source Stirrup harness with code execution and an image viewing tool, then reports pass cubed (written pass^3): the share of tasks a model answers correctly on all three repeats. When we read the VIALS-AA leaderboard on November 8, 2026, Gemini 4 Carbon (high) led at 64%, ahead of GPT-6 Astra (max) at 63% and Claude Opus 5.5 (max, with default fallback) at 61%. This guide explains what that score measures, why it will not match Handshake's own chart, and where it can mislead. We are not picking a best model.
What VIALS-AA measures
VIALS stands for Visual Interpretation of Artifacts in the Life Sciences and sits inside Handshake AI's ATLAS suite. The upstream paper (arXiv:2608.21357v2, dated September 19, 2026) argues that today's vision language models can describe everyday photos fluently and still misread the figures scientists actually use: the wrong gel band, the wrong flow gate, or a contact that is not there in a protein ligand view.
Each of the 161 tasks pairs one or more images with an open ended question and a short expert reference answer. A correct response can use different wording or units as long as it matches the reference scientifically. Artificial Analysis counts 174 source images. Handshake's dataset card says most tasks have one image, four tasks bundle two to eight images, and five count style tasks publish an accepted numeric range.
Handshake reports that 31 contributors built and reviewed the tasks (87% with doctoral training, 74% with more than six years of professional research experience). A second expert, and often a third, answered each task independently without seeing the original answer (about 16 minutes on average). A task stayed in only if those answers agreed or the experts later reached consensus. In a blinded survey in the paper, reviewers classed 63% of VIALS artifacts as resembling professional work, against 22% of the biology and chemistry image questions from Humanity's Last Exam.
The current public dataset is version 2.0.0, tagged on October 30, 2026 after three tasks were updated; paper results used version 1.0.0 (September 15, 2026). Artificial Analysis says it evaluates the 161 task set Handshake released but does not name the revision on the pages we read, so cite the board and the date. The dataset is CC BY 4.0, and Handshake's runner is MIT licensed on GitHub.
The eight artifact domains
Handshake assigns each task one primary domain. Shares below come from Table 1 in the paper:
| Domain | Share | What the images look like |
|---|---|---|
| Phylogenetics | 14.3% | Rectangular and circular trees, clade and taxon labels |
| Flow cytometry | 14.3% | Gating plots, FMO overlays, histograms and multi parameter panels |
| Blotting | 14.3% | Western blots, SDS-PAGE, 2D gels, lane and band comparisons |
| Cell counting and quantification | 14.3% | Fluorescence images, colonies, plaques, smears and confluence views |
| Plasmid maps | 13.0% | Circular and linear maps, restriction sites and feature annotations |
| Protein structure and binding | 13.0% | Ligand diagrams, active site views and residue contacts |
| Molecular structure | 12.4% | Small molecule skeletons, substituents and transformations |
| Other | 4.3% | Pedigrees, lateral flow assays and related figures |
SDS-PAGE and 2D gels are official method names. On VIALS-AA, Gemini 4 Carbon (high) led or shared the lead on plasmid maps, phylogenetics and flow cytometry (sharing flow cytometry with Gemini 4 Argon). Claude Opus 5.5 (max) led on blotting, Claude Sonnet 5.5 (max) on cell counting, GPT-6.1 Terra (max) on protein structure and binding, and GPT-6 Astra (max) and GPT-6.1 Sol (max) shared molecular structure.
How the VIALS-AA score is calculated
Artificial Analysis does not copy Handshake's published numbers. It reruns the public tasks under one harness so models face the same tools and the same grading rule.
- Harness. Models run in Stirrup, Artificial Analysis's open source reference harness. They receive the question and the image file paths. The sandbox offers Python code execution with no network access, plus an image viewing tool for vision capable models.
- Repeats. Each model attempts every task three times, for up to 483 scored rollouts per model.
- Headline metric. Pass cubed is the share of tasks answered correctly on all three repeats. The board also shows average correctness across those repeats, and pass@3 (correct on at least one of the three). Artificial Analysis uses pass cubed because a result that changes from run to run is a poor basis for a scientific decision.
- Grading. GPT-6 Luna (medium) grades each final answer with Handshake's upstream judge prompts. When needed, a second judge call includes the task images. A repeat with no submitted answer, or with no
ANSWER:line, scores zero. Refusals also score zero. - Cost. Cost per task averages input, cache, reasoning and answer token prices across the three repeats. Judge tokens are excluded.
That setup is an agentic, tool assisted reading of the figures, not a single glance at the image, which is why VIALS-AA numbers and Handshake's own chart should not be mixed. For how model graders can go wrong even with a short reference answer, see LLM as a judge.
Why Handshake's chart and VIALS-AA disagree
The paper's main setting is a direct vision language call: the model sees the images and the question in one request, with no code tools. On version 1.0.0, the best models Handshake reported (GPT-5.6 Sol and Gemini 3.7 Flash) reached only 26.5% average accuracy across three rollouts, and pass cubed stayed under 17%. About 83% to 93% of errors were in reading the figure; visual quantification (miscounting cells, colonies or bands) was the most common mode, at 28.2% to 41.8% of errors.
The paper then gave the same models code execution so they could crop, zoom and measure. Accuracy rose by as much as 43.3 points (Kimi K3 in OpenCode), and the best reported agent configuration reached 65.8% average accuracy (Claude Opus 5 in OpenCode), at 10 to 432 times the token cost of the direct call.
Handshake's public VIALS page now shows a live accuracy versus cost chart with newer models. When we read it on November 8, 2026, GPT-6 Astra led at 49.9% accuracy and about $0.0478 per attempt, with GPT-6.1 Sol at 49.1% and about $0.008. The page does not spell out, in the text we could verify, whether that chart is the direct one shot setting or a later agent run, so treat it as Handshake's own score.
VIALS-AA is the independent agentic rerun: one harness (Stirrup), one judge model (GPT-6 Luna, medium) and pass cubed as the headline. GPT-6 Astra (max) at 63% pass cubed on VIALS-AA and GPT-6 Astra at 49.9% accuracy on Handshake's chart are not two readings of the same test. Handshake validated its original judge (GPT-5-mini) against human grades on 4,685 responses and reported 99.9% agreement. Artificial Analysis uses those prompts with a different judge model.
First results on the VIALS-AA leaderboard
Figures below come from the November 6 launch article and from the board as we read it on November 8, 2026. The leaders had not moved in that interval.
| Model and setting | Pass cubed |
|---|---|
| Gemini 4 Carbon (high) | 64% |
| GPT-6 Astra (max) | 63% |
| Claude Opus 5.5 (max, default fallback) | 61% |
| Claude Sonnet 5.5 (max, with fallback) | 60% |
| GPT-6.1 Sol (max) | 59% |
| Kimi K3.1 Pro (max) | 45% |
Gemini 4 Carbon (high) answered all three repeats correctly on 103 of 161 tasks, two more than GPT-6 Astra (max). The launch article's score versus cost Pareto set is Gemini 4 Carbon (high), GPT-6 Luna (max), GPT-6.1 Sol (max), GPT-6.1 Terra (max) and GPT-6 Astra (max).
Consistency is the point of the headline. Kimi K3.1 Pro (max) and GPT-6 Astra (max) each solved around three quarters of the tasks at least once, but pass cubed split them 45% to 63%. Leading open weight models (Kimi K3.1 Pro, MiMo-V2.6-Pro and MiniMax-M3.1-Flash) solved 114 to 119 tasks at least once, matching the seven top closed models (111 to 118), yet they passed all three repeats on only 48 to 73 tasks, against 95 to 103. If you only quote pass@3, the open weight gap almost disappears.
Cost per task ranged from $0.02 for GPT-6 Luna (max) to $1.77 for Grok 4.7 (xhigh), about 99 times. GPT-6.1 Sol (max) scored 59% at $0.06 per task, about one thirtieth the cost of Claude Opus 5.5 (max, with fallback) for a similar 61%. Claude Sonnet 5.5 (max) scored 60% at about 18 times Sol's cost.
These are reading scores, not buying advice. VIALS-AA is not one of the nine evaluations inside the Artificial Analysis Intelligence Index v5; see also LLM leaderboards compared.
Why more inspection does not always help
Across 9,177 recorded rollouts of the original 19 model set, Artificial Analysis classified tool calls by their main activity: 38.3% were image views and 47.3% prepared or measured images. Image measurement was only 5% to 6% of calls for GPT-6 Astra (max) and GPT-6.1 Sol (max), against 25% to 27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max). Claude Sonnet 5.5 (max, with fallback) and GPT-6.1 Sol (max) scored 60% and 59%, but Sonnet made 3.6 times as many tool calls per task.
The launch article walks through Task 1, "Protein spot 22," which asks for the molecular weight range of a labeled spot on a 2D gel. The official reference is the range 40 to 50 kDa. GPT-6 Astra (max) enlarged the image once and answered correctly in six turns. Claude Sonnet 5.5 (max) cropped, drew guides and interpolated the ladder, finishing correctly in 18 turns. Grok 4.7 (xhigh) used 27 turns and 20 image views, then answered 50 to 70 kDa. Artificial Analysis's reading is that Grok anchored on the printed label and the arrow tip rather than on the faint protein spot.
The paper's failure analysis helps explain the pattern: most direct vision errors were already in reading the figure, not in later arithmetic. Tools help when they put attention on the right pixels, not when the model measures the label instead of the spot.
Limits to keep in mind
- Pass cubed is strict by design. Right twice and wrong once earns no credit on that task.
- 161 tasks is a small sample. One domain holds about 7 to 23 items, so a few flips can move a domain lead. The paper calls the set limited in scale.
- The headline hides domain spikes. Quote the domain that matches your artifacts.
- Harness and judge are choices. Stirrup, GPT-6 Luna (medium) as judge and the required answer format all shape the score.
- Version drift. Version 2.0.0 changed three tasks on October 30, 2026. Paper numbers, Handshake's live chart and VIALS-AA should each be cited with a date.
- Vision work, not a wet lab agent suite. VIALS asks a visual question about provided artifacts, not a full experimental workflow from raw instrument files.
- No model pick. A 64% pass cubed score means the leader still fails more than a third of the set when you require three consistent answers. Shortlist from the board, then test on your own images; see how to evaluate LLM model performance.
For specialist work tests on economic tasks rather than scientific figures, see the GDPx benchmark and GDPval, and LLM benchmarks explained for the broader map.
FAQ
What is VIALS-AA?
VIALS-AA is Artificial Analysis's independent implementation of Handshake AI's VIALS benchmark: 161 scientific image tasks with a short question about the evidence in each figure. The public board launched on November 6, 2026.
What does pass cubed mean on VIALS-AA?
Pass cubed (pass^3) is the share of tasks a model answers correctly on all three independent repeats. Average of 3 is mean correctness across those repeats, and pass@3 is success on at least one repeat. Artificial Analysis headlines pass cubed because inconsistent answers are risky in scientific work.
Why do Handshake VIALS scores and VIALS-AA scores disagree?
They are different evaluations. Handshake reports a direct one shot vision setting, a separate tool assisted setting, and a live accuracy chart. VIALS-AA reruns the public tasks in Stirrup, grades with GPT-6 Luna (medium), and headlines pass cubed. A 49.9% Handshake accuracy and a 63% VIALS-AA pass cubed for GPT-6 Astra are not interchangeable.
Can I run VIALS myself?
Yes, the public tasks. The dataset is on Hugging Face as handshake-ai-research/VIALS (CC BY 4.0), and Handshake's MIT licensed runner is on GitHub. Matching VIALS-AA also requires Stirrup, the judge model and the rest of Artificial Analysis's configuration.
Is VIALS-AA part of the Artificial Analysis Intelligence Index?
No. The Intelligence Index v5 averages nine other evaluations. VIALS-AA is a standalone board.
How is VIALS different from HLE image questions?
Humanity's Last Exam includes some biology and chemistry image items, but Handshake's expert survey found those items look more like academic or exam questions. VIALS asks for the kind of reading a scientist would do at the bench on gels, flow plots, plasmid maps and structures.


