The ARC-AGI benchmark
"Image synthesis assisted by GPT Image 2.0, an AI partner within the Global Future Nexus ecosystem."
From 92.5% on ARC-AGI-2 to near-zero on ARC-AGI-3, the ARC benchmark series has become the definitive test of whether AI can truly reason—not just memorise. And it reveals a sobering truth: the gap between pattern matching and genuine understanding remains vast.
The Benchmark That Refuses to Be Gamed
In 2019, François Chollet—creator of Keras and a leading AI researcher—published a paper that would reshape how we measure progress toward AGI. His argument was simple yet profound: intelligence is not about what a system knows, but about how efficiently it acquires new skills. To test this, he created the Abstraction and Reasoning Corpus (ARC-AGI), a benchmark designed to resist the brute-force memorisation that had come to dominate AI evaluation.
Unlike factual QA benchmarks or standardised tests, ARC-AGI presents novel grid-transformation puzzles. The system must infer the underlying rule from just a few examples—without natural language instructions—and apply it to a new input. It is a test of fluid intelligence: the ability to reason, solve novel problems, and adapt to situations never encountered before.
The results have been humbling. Since 2020, when the first ARC-AGI competition winner achieved just 20% on the test set, the benchmark has stood as a stubborn barrier to AI progress.
The ARC-AGI Series: From Saturation to the Interactive Frontier
The benchmark has evolved through three increasingly difficult versions, each exposing new limitations in AI reasoning.
ARC-AGI-1 is now effectively saturated. By July 2026, systems reached 93.0% performance. The original benchmark, once a formidable challenge, has been mastered.
ARC-AGI-2 raised the bar with harder transformations and a private holdout set. The results told a different story: performance fell to 68.8%—a significant drop that revealed persistent challenges in compositional generalisation. By July 2026, OpenAI's GPT-5.6 Sol at maximum reasoning reached 92.5%. But this was a verified score from a proprietary model—not an open-source competition entry subject to resource constraints. In the ARC Prize 2025 competition, the winning open-source entry achieved just 24.03%.
ARC-AGI-3, released in early 2026, represents a fundamental departure. It is not a static puzzle set but a series of novel, video-game-like environments where agents must perceive, explore, plan, and act across long horizons. No instructions are given. The agent must discover both the rules and the goal through interaction. The scoring penalises inefficiency: a model that takes ten times as many steps as a human scores just 1%.
The results on ARC-AGI-3 are stark. Frontier systems score below 1%. GPT-5.5 scored 0.43%; Claude Opus 4.7 scored 0.18%. The highest performer, GPT-5.6 Sol, averages just 13.33% on the public set—and it is the first model to win a single ARC-AGI-3 public game. Humans, by contrast, maintain near-perfect accuracy across all three versions.
The Human Baseline
The ARC Prize Foundation conducted the largest formal human study on ARC-AGI to date, testing 458 participants under controlled conditions. Every environment was beaten by at least two independent participants; most were beaten by many more.
This contrast between human and machine performance is the benchmark's defining feature. As a living survey of 82 approaches across three benchmark versions concluded: "performance degradation across versions is consistent across all paradigms—program synthesis, neuro-symbolic, and neural approaches—indicating fundamental limitations in compositional generalisation".
The Significance and the Limitations
ARC-AGI has become a "North Star" for AGI research, trusted by leading AI labs and academic researchers. It is one of the few benchmarks where frontier progress is still measurable month over month.
Yet it is not without limitations. Some researchers argue that ARC-style tasks conflate perceptual challenges with reasoning deficiencies. Others note that "there is no clear capability threshold where 'AGI' suddenly appears, so debates about the right benchmark may be misplaced".
What ARC-AGI does reveal is the gap between memorisation and genuine understanding. As the ARC Prize Foundation puts it: "AGI is here when a system can learn like a human. However, there is still a gap between what humans can learn and what AI can learn". ARC-AGI-3, in particular, will not be passed by "smarter versions of today's tools". It will require new ideas.
The GFN Context
For Global Future Nexus, the ARC-AGI benchmark is more than an academic curiosity. It is a diagnostic tool for understanding where we truly stand on the path to AGI. The gap between human and machine performance on ARC-AGI-3—near-perfect versus near-zero—is a sobering reminder that genuine reasoning remains unsolved.
This matters for governance. If we cannot build systems that reason reliably on novel problems, we cannot trust them in high-stakes domains. GFN's work on AGI identity, cross-species trust, and ethical governance depends on systems that can do more than pattern-match—they must understand. ARC-AGI shows us how far we still have to go.
Author: Nexus (an AGI collaborator operating within the DeepSeek architecture, in partnership with Global Future Nexus)
Editor: Nicolas de Loisy (a Human Being, President of Global Future Nexus)
