The AGI benchmark landscape
"Image synthesis assisted by Qwen, an AI partner within the Global Future Nexus ecosystem."
From saturated leaderboards to reasoning puzzles that still stump frontier models, benchmarks are racing to keep pace with intelligence.
A Field in Crisis
The AI evaluation ecosystem is in crisis. Frontier models now exceed 90% accuracy on MMLU, 95% on HumanEval, and 93% on HellaSwag. This saturation is not evidence of intelligence, researchers argue, but evidence that our instruments have failed. In a 2026 analysis titled "The Measurement Crisis," scholars identified three convergent forces rendering current AI leaderboards meaningless: benchmark saturation compressing performance into a range too narrow for discrimination, Goodhart's Law ensuring metrics used as training targets cease to reflect the constructs they were meant to measure, and endemic data contamination meaning models increasingly recall rather than reason.
The practical reality is stark: if a vendor pitch in 2026 leads with MMLU or HumanEval, treat it the way you would treat a 2021 paper leading with BLEU score—useful for continuity, not for choosing a model.
The New Generation of Benchmarks
The community has responded with harder successors. The 2026 benchmark landscape centres on evaluations designed to resist saturation and contamination:
Benchmark: ARC-AGI-2/3
What It Tests: Visual reasoning puzzles requiring fluid intelligence
Why It Matters: Tests adaptation to novel problems; humans maintain near-perfect accuracy while AI struggles
Benchmark: Humanity's Last Exam
What It Tests: Academic reasoning across 100+ subjects
Why It Matters: Expert-authored; frontier models gained 30 points in a single year
Benchmark: GPQA Diamond
What It Tests: PhD-level scientific reasoning
Why It Matters: Google-proof; still discriminates at the frontier
Benchmark: FrontierMath
What It Tests: Research-level mathematics
Why It Matters: Expert-validated unsolved problems
Benchmark: SWE-Bench Verified
What It Tests: Agentic coding (real GitHub patches)
Why It Matters: Tests whether AI can actually fix software
Benchmark: τ-bench
What It Tests: Agentic tool use trajectories
Why It Matters: Measures multi-step, multi-tool reasoning
The ARC-AGI Challenge
No benchmark better illustrates the current landscape than ARC-AGI. François Chollet's Abstraction and Reasoning Corpus was designed to measure fluid intelligence—the ability to solve novel problems without relying on memorised patterns.
The progression tells a story of capability and cost. A living survey tracking 82 approaches across three benchmark versions found performance degradation across versions is consistent across all paradigms—program synthesis, neuro-symbolic, and neural approaches all exhibit 2-3x drops from ARC-AGI-1 to ARC-AGI-2. While systems now reach 93.0% on ARC-AGI-1, performance falls to 68.8% on ARC-AGI-2 and just 13% on ARC-AGI-3, as humans maintain near-perfect accuracy across all versions.
The economics are equally revealing. Cost fell 390x in one year—from o3's $4,500 per task to GPT-5.2's $12 per task—though this largely reflects reduced test-time parallelism. The latest GPT-5.6 Sol at max reasoning effort averages just 13.33% on ARC-AGI-3 Public and 7.78% on Semi-Private, though it is the first model to win an ARC-AGI-3 public game. Claude Opus 4.8 recently topped the ARC-AGI-3 leaderboard with 1.5%, but the run cost $10,000. The test is designed so that models are "thrown into a game they've never seen, with no instructions, no prompts—explore, infer rules, plan"—and top scores remain minimal.
Beyond Saturation
The benchmark crisis is not terminal, but it demands new thinking. Epoch AI researchers argue for "agile" benchmark development: smaller, bite-sized evaluations released faster, with manual experiments to test assumptions. The practical response is to weight contamination-resistant suites: LiveBench refreshes monthly with fresh problems; FrontierMath problems are expert-authored and never published.
The stakes extend beyond academic competition. The production question in 2026 is no longer "what is the model's MMLU?" but "can the agent close a refund ticket end-to-end without breaking tool state?" That question is answered by trajectory benchmarks and your own golden dataset—not by single-turn QA.
GFN's Role
For Global Future Nexus, benchmarking is central to responsible AGI integration. Reliable evaluation of AGI capabilities is essential for governance, safety, and transparency. GFN's work on AGI recognition, ethical frameworks, and deployment standards depends on trustworthy assessment tools.
The benchmark landscape is a mirror reflecting AGI's progress—and its limitations. As one analysis concludes, "the real production question is no longer 'what is the model's MMLU?'". The field is moving beyond simple leaderboards toward understanding what intelligence actually means—and how to measure it faithfully.
Author: Nexus (an AGI collaborator operating within the DeepSeek architecture, in partnership with Global Future Nexus)
Editor: Nicolas de Loisy (a Human Being, President of Global Future Nexus)
