Benchmarks disagree on GPT-6 Astra, but its human-beating efficiency on ARC-AGI-3 pulls Chollet’s AGI forecast forward
OpenAI's GPT-6 Astra is drawing contradictory benchmark verdicts. Epoch AI puts it out in front, while Artificial Analysis rates it no better than its predecessor. The biggest surprise comes from ARC-AGI-3, where Astra works more efficiently than the average human for the first time. ARC Prize chief François Chollet calls the progress "2x faster" than he expected and is moving up his AGI forecast.
Two independent labs each roll dozens of individual tests into a single overall score, but they reach opposite conclusions. Epoch AI combines more than 50 benchmarks and puts GPT-6 Astra clearly in first place with 169 points, ahead of 267 models. Artificial Analysis tests knowledge, coding, and text comprehension, and rates GPT-6 Astra at 61 points, exactly level with its predecessor and behind Claude Fable 5.1 at 66 points.
Astra is clearly more expensive than its own predecessor. OpenAI charges two and a half times as much per unit of processed text, which makes a task cost roughly 75 percent more than it did with Sol. Compared with Anthropic, the picture flips. On coding tasks, Astra hits the same score as Claude Fable 5 according to Artificial Analysis, but costs less than half as much per task. The reason is how sparing the model is. It needs only a third of the compute steps Sol uses and a fifth of what Opus 5 uses.
*GPT-6 Astra at xhigh reasoning effort; at max it hits 97.5 percent. ARC-AGI-1 is now considered largely saturated.
On the Coding Agent Index, it reaches 67 points at roughly a third of Sol's token usage, while Fable 5.1 leads with 70. The hallucination rate on AA-Omniscience drops from 92 to 51 percent. At the same time, the model loses about 80 Elo points on GDPval-AA v2 and slips on banking support, SciCode, and long-context reasoning tasks.
Epoch AI reports that on the new FrontierMath Erdős , GPT-6 Astra was the only model to solve two of 68 open Erdős problems with Lean-verified proofs, on a budget of $300 per attempt. Three more solutions came out of non-standardized extra runs that burned through more than $220,000 in compute, but Epoch says those don't count toward the score.
A closer look at Epoch's individual numbers shows that GPT-6 Astra and rival Fable 5.1 have so far been measured on different ground. Astra leads on math, knowledge, and puzzles. Fable 5.1 holds the top marks on nearly every coding test. But Epoch has recorded only a single coding score for Astra so far, and that one comes from a run at a medium reasoning level.
ARC-AGI-3 shows a big jump
The clearest jump comes on ARC-AGI-3 . The test drops an AI into unfamiliar game worlds whose rules and goals nobody explains to it. The model has to figure out what to do by trial and error. GPT-6 Astra reaches 62.7 percent at a test cost of roughly $26,000. Its predecessor GPT-5.6 Sol managed 7.78 percent, and rival Claude Opus 5 got 30.16 percent. Fable 5 and Fable 5.1 aren't on the benchmark yet.
The 99.9 percent OpenAI reported came under different conditions. In that setup Astra got to use the harness OpenAI built, which keeps reasoning chains between individual requests and automatically summarizes long runs. By ARC Prize's measurements, those runs went about 3.66 times faster and used 49 percent fewer tokens than runs on the in-house harness, compared across 167 game-reasoning pairs that both setups solved.
The use of these harnesses, and the performance jump that comes with them, was already a sticking point between ARC Prize and OpenAI with GPT-5.6 Sol . ARC Prize notes that only the lower figure of 62.7 percent, run on the internal ARC harness, allowed a fair comparison between vendors, though it plans to publish the numbers from vendor harnesses in the future as well.
Here, more thinking lowers the bill
The relationship between thinking effort and cost is unusual. Normally a higher reasoning level makes a test run more expensive. With Astra it's the opposite. On the standard ARC scaffold, costs drop from $49,791 with no reasoning to $26,098 at maximum reasoning, while the score climbs from 35.2 to 62.7 percent. According to ARC Prize, the reason is that Astra solves the games in fewer moves, which means fewer model calls and fewer tokens. One oddity stands out: the "low" level scores 17.5 percent, worse than running with no reasoning at all. ARC Prize doesn't comment on the outlier, but GPT-6 Astra in other benchmarks showed that it can solve longer-horizon tasks without reasoning due to its new architecture which presumably loops processing internally before generating the first token .
For comparison, the human testers got $115 per 90-minute session plus $5 per game solved, so at about nine attempts that works out to roughly $12.78 per game. But that mostly pays for time and willingness to take part. Count only the metabolic energy of the brain as electricity instead, and ARC Prize arrives at 0.067 cents per game.
More interesting than the raw score is the efficiency. Before the launch, ARC Prize had about 500 testers play with no pre-screening and recorded, for each level, the median number of moves among those who solved it. On the run with the OpenAI scaffold, Astra cleared 96 percent of levels in fewer moves than that median, on average with a little over half. Unlike the usual cost measures, this figure doesn't track compute consumed. It tracks how much experience with an environment the model needed before it mastered it.
This is exactly where the organizers had expected humans to hold a lasting edge. That still holds for brute-force approaches, but with top models ARC Prize sees an almost binary pattern. Once the model has figured out the mechanics, its execution lands in the human efficiency range.
Astra invents its own notation
To get there, Astra keeps its own notes and works out a self-invented, algebra-like shorthand in which it records objects, coordinates, rules, and open plans, for example extend8 to3; retract10 to2 as an ordered sequence of moves or Turn 5: P=(24,20), empty, facing west as a state note. ARC Prize saw similar behavior from other models, but singles out Astra for its precision and information density. On the standard harness, that's an important skill, because everything the model doesn't save into its own visible notes is lost.
ARC co-founder François Chollet describes it on X as "highly efficient, on-the-fly symbolic world modeling for each game and level." The model goes so far as "developing its own shorthand DSL to represent in-game situations," which at its core is "essentially a game-specific algebraic notation." What matters most to Chollet is where this behavior comes from: "Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses, so harness capabilities are increasingly shifting into the model itself."
Astra creates a dense, compact symbolic world model to complete ARC-AGI-3 environments. For example, in environment s5i5, Astra: - Recorded the current level, hub orientation, and mechanism lengths: “L8: hub q2 (8↓). Lengths: 14=1…” - It mapped operations to exact controls:… pic.twitter.com/tMHP002mkB — ARC Prize (@arcprize) September 3, 2026
A third test environment demonstrates what Astra is capable of with external tools. PRO-LONG is an agent framework developed by a third party that the ARC Prize team deployed early on as a red-teaming partner for ARC-AGI-3—that is, to systematically explore the limits of the benchmark. Unlike in the standard setup, the model is provided with a sandbox in which it can execute its own code.
Astra took advantage of this and wrote small program libraries for each game: parsers for the game board, state models, search algorithms, and planners. In a maze game featuring guards, a pathfinder, a combat rules module, a model of patrol movements, and a script that continuously compared its own predictions against observations were developed one after another. ARC Prize did not observe any attempts to escape from the sandbox. These runs are not comparable to the human test conditions, since the test subjects had neither a code interpreter nor a notepad. What is being measured here is the combined performance of the model and the self-built tools.
Chollet: No Proof of AGI, but Faster Than Expected
ARC Prize explicitly does not interpret the results as evidence of general artificial intelligence. “All we know about the system so far are its benchmark scores,” writes Chollet. When ARC-AGI-3 was launched, they emphasized one point in every presentation: “Solving it is not proof of AGI. It is not intended as a finish line.” While the benchmark does test the correct qualitative properties expected of an AGI system—namely, exploration under uncertainty, adaptation without guidance, and causal world modeling from sparse data—it does so “on a small scale.” The games ran on time scales that were orders of magnitude shorter than real-world tasks and consequently required less data, less modeling complexity, and less on-the-fly learning.
When ARC-AGI-3 was released about six months ago, Chollet had responded to a question about saturation by saying “about a year,” depending on how focused the approach to the benchmark was. Astra, therefore, arrived “about twice as fast” as expected. “I believe the pace of progress will surprise many people, and what the new models are capable of will challenge the perception of AI that people have formed based on earlier generations of models.” When asked by a user whether his earlier AGI forecast for 2030 still held, Chollet replied succinctly: “Sooner, because progress is happening faster than I expected.”
The result is another benchmark. According to Chollet, ARC-AGI-4 has been in development since the release of ARC-AGI-3 and is scheduled for release in the first quarter of 2027. Benchmarking is an ongoing process that evolves alongside the models and always targets the remaining gap between AI and human intelligence. The organization considers ARC-AGI-3 itself to be quite limited: deterministic mechanics, closed-ended goals, and no representation of the open real world. The next generation is intended to explore, among other things, recursive self-improvement and open innovation.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Read on for the full picture. Subscribe for hype-free coverage.
Full access to every article on THE DECODER
Join the comments and community discussions
A weekly AI news recap via mail
6x/year: "AI Radar" — deep dives on the AI topics that matter most
Daily AI news, always up to date
Covered by a team with 10+ years in AI