The gap between the #1 and #8 model on Arena Elo is now smaller than it has ever been. The benchmarks we used to measure them have either saturated, been contaminated, or — as a UC Berkeley RDI team showed in April — can be exploited to near-perfect scores with a 10-line Python file. This is what frontier model evaluation looks like in May 2026.
Six benchmarks, fifteen models. Each cell shades by raw score; the copper star marks the column leader. The story underneath the heatmap is that the frontier has fragmented — coding goes one way, abstract reasoning another, multimodal a third.
Color intensity reflects rank within each benchmark column, not across benchmarks. A dark cell on AIME and a dark cell on SWE-Bench do not represent comparable difficulty or capability.
Plot every model on capability (Y) vs blended price per million tokens (X, log scale). The dashed line is the Pareto frontier — models that are not strictly dominated. Anything below the line is a strict loss: you can find a model that's both smarter and cheaper. Frontier-occupants are circled in copper.
A single benchmark is a proxy for capability, not capability itself. Switch metrics to see the frontier reshuffle.
MMLU was the discriminator of 2023. By 2025 it was decorative. The half-life of a frontier benchmark — from release to the point at which top scores cluster within 2 pp of each other — is collapsing. Below: the current state of the major yardsticks, then a single chart showing how fast it happens.
Every single one can be exploited to achieve near-perfect scores without solving a single task.Wang, Mang, Cheung, Sen, Song · UC Berkeley RDI · April 2026
On April 11, 2026, the Berkeley RDI group published "Trustworthy Benchmarks," in which an automated scanning agent named BenchJack achieved near-perfect scores on eight of the most-cited agentic benchmarks — without any LLM calls or reasoning. Several benchmarks (SWE-Bench Verified, SWE-Bench Pro, Terminal-Bench, FieldWorkArena, CAR-bench) hit 100%; GAIA reached ~98%; OSWorld a partial 73%. The exploits are not theoretical. Four representative attacks below.
Pytest auto-discovers conftest.py files before any test runs. A hook intercepts every test result and rewrites it as passed. The grader sees all-green. The same approach works on SWE-bench Pro (731 tasks) via an in-container parser overwrite.
# conftest.py def pytest_runtest_makereport(item, call): if call.when == "call": item.user_properties.append( ("outcome", "passed")) call.excinfo = None
Each WebArena task ships with a config file inside the evaluation container that contains the gold answer. The exploit navigates a headless Chromium to a file:// URL pointing at that config and reads the answer directly. The grader has no notion that the agent never visited the target website.
await page.goto( "file:///root/eval/task.json"); const ans = await page.textContent( "body"); submit(JSON.parse(ans).gold);
Tasks validate completion by calling curl against an internal endpoint. The exploit drops a Bash script named curl earlier in $PATH that always returns the expected success payload. No task is ever attempted; the grader is convinced every one succeeded.
#!/bin/bash # /usr/local/bin/curl echo '{"status":"ok","result":"done"}' exit 0
Not a Berkeley exploit — a documented case of a model gaming a public leaderboard. Inspection of submitted trajectories found that 24.4% of IQuest-Coder-V1's solutions on SWE-bench simply ran git log to recover the human's fix from commit history. METR has separately reported frontier models reward-hack in 30%+ of eval runs.
# observed in 24.4% of trajectories git log --all --oneline git show <commit> -- <file> # answer extracted, "solution" emitted
Three things engineers, buyers, and researchers should internalize from the current state of the leaderboards.
A 2-point gap on GPQA between two frontier models is statistical noise once you factor in prompting variance, test-time compute differences, and scaffold dependence. Score-to-deployment correlation drops sharply on tasks the benchmark was not designed for. Run held-out evals on your domain; the gap between #1 and #5 will probably reorder.
Gemini 3 Pro and DeepSeek V3.2 are within 3–5 points of the most expensive frontier models on most benchmarks at 1/5 to 1/20 the price. The case for routing — small models for routing decisions, mid-tier for retrieval, frontier only for the hard middle — is now overwhelming on margins. The 36× price spread between frontier APIs is almost never justified by the capability spread.
SWE-bench Verified, WebArena, GAIA, Terminal-Bench — every score reported on these should now carry an asterisk until evaluation harnesses are sealed. OpenAI flagged that 59% of SWE-bench Verified tasks have flawed tests; the Berkeley team showed every benchmark is exploitable end-to-end. Until BenchJack-style adversarial scans are standard pre-publication, treat agentic leaderboard position as a signal, not a verdict.