Scaling Pre-Training Will Not Reach AGI, Says ARC-AGI Creator

The entire AI industry runs on one bet: make models bigger, and intelligence emerges. Francois Chollet has been making the counter-argument for years, and the data keeps proving him right.

50,000x Scale-Up, Same Result

Chollet's core evidence is the ARC-AGI benchmark itself. Base language models scaled 50,000x in compute and parameters between 2020 and 2026. Their scores on ARC-AGI V1 barely moved.

"Base models were scoring extremely low on ARC V1 like sub 10% basically...even though in the meantime we had scaled up these models by 50,000x. It was really telling you that scaling up pre-training alone was not going to crack the benchmark." — [Francois Chollet, 17:30](https://www.youtube.com/watch?v=k2ZLQC8P7dc&t=1050)

Pattern matching scales. Memorization scales. Fluid intelligence — the ability to solve problems you have never seen before — does not. Pre-training is compression of the training set. More data and more parameters give you better compression, not a fundamentally different kind of cognition.

The Harness Illusion

ARC-AGI V2 was eventually solved, but not by smarter models. It was solved by wrapping models in RL loops: generate a solution, verify it, finetune on the successes, repeat. This is the same paradigm behind coding agents and every "agentic" system in production today.

Chollet draws a sharp line between usefulness and intelligence. The models are more useful. They are not more intelligent.

"The models do not have higher fluid intelligence per se. They do not have a higher IQ so to speak. It is just that they are way better trained." — [Francois Chollet, 22:42](https://www.youtube.com/watch?v=k2ZLQC8P7dc&t=1362)

The harness is doing the cognitive heavy lifting. The model is the engine, but the engineering scaffolding around it — the retry logic, the tool use, the verification step — is where the intelligence actually lives. And humans built that scaffolding.

"The fact that you need humans to engineer these harnesses is also a sign that we are short of AGI today. If we had AGI, the AI would just make its own harness." — [Francois Chollet, 24:47](https://www.youtube.com/watch?v=k2ZLQC8P7dc&t=1487)

This is the uncomfortable truth beneath the current agent boom. Every SOTA result on [SWE-bench Verified](https://benchmark.space/benchmark/swe-bench-verified) or [OSWorld](https://benchmark.space/benchmark/osworld) depends on hand-crafted scaffolding. Remove the harness and the model's performance collapses.

ARC-AGI 3: Measuring Agentic Intelligence

Chollet's answer is ARC-AGI V3, which targets exactly the capability gap that current benchmarks miss. V1 and V2 presented the model with a puzzle and expected an answer. V3 is interactive. The model must explore an environment, set its own goals, and figure out what to do without instructions.

"ARC V3 is completely different. We are trying to measure agentic intelligence. It is interactive, it is active -- the data is not provided to you, you must go get it." — [Francois Chollet, 25:49](https://www.youtube.com/watch?v=k2ZLQC8P7dc&t=1549)

The early results are stark. Every frontier model scores below 0.5% on [ARC-AGI 3](https://benchmark.space/benchmark/arc-agi-3):

A model that scores 85% on [ARC-AGI 2](https://benchmark.space/benchmark/arc-agi-2) scores 0.2% on V3. The gap is not incremental. It is categorical. These models can solve problems when the problems are handed to them. They cannot find the problems on their own.

The 10K-Line Hypothesis

Chollet's most provocative claim is about what AGI will actually look like when it arrives. It will not be a trillion-parameter model. It will be a small, elegant program.

"When we create AGI, retrospectively it will turn out that it is a code base that is less than 10,000 lines of code, and that if you had known about it back in the 1980s you could have done AGI back then using the compute resources available back then." — [Francois Chollet, 36:05](https://www.youtube.com/watch?v=k2ZLQC8P7dc&t=2165)

The insight is that intelligence is not about compute or data. It is about the right algorithm. A 1980s machine could run 10,000 lines of code. What it lacked was not FLOPs but the right program. We still lack it.

The Takeaway

The AI industry has conflated two things: competence on known tasks, and the ability to handle genuinely novel situations. The first scales. The second does not. ARC-AGI 3 is the first mainstream benchmark that forces the distinction, and the scores confirm it. Until a model can explore, plan, and self-scaffold in an environment it has never seen, calling it AGI is premature. The algorithm matters more than the parameters.

Model	ARC-AGI 3 Score	ARC-AGI 2 Score
Gemini 3.1 Pro Preview	0.4%	—
GPT-5.4	0.3%	73.3%
Claude Opus 4.6	0.2%	85.0%
Grok 4.20	0.0%	—