The benchmark numbers look unambiguous. Claude Mythos hits 93.9% on [SWE-bench Verified](https://benchmark.space/benchmark/swe-bench-verified). GPT-5.4 clears 80%. By the scoreboard, AI coding is solved. Jeremy Howard disagrees — and his argument is not about the numbers. It is about what the numbers fail to measure.
Howard draws a hard line between coding and software engineering. Coding, in his framing, is style transfer — interpolation within training data. Software engineering is designing novel abstractions to solve problems no one has solved before. LLMs are competent at the first and show no empirical trend toward the second.
"There's no current empirical data to suggest that LLMs are gaining any competency at software engineering. Every time you look at a piece of software engineering they've done like the browser Cursor created or the C compiler Anthropic created, they're very obvious copies of things that already exist." — [Jeremy Howard, 59:39](https://www.youtube.com/watch?v=dHBEQ-Ryo24&t=3579)
This is a pointed critique. When Anthropic announced their C compiler and Cursor shipped a browser, both were heralded as evidence of genuine engineering capability. Howard calls them what they are: interpolations of existing code in the training set. The abstractions already existed. The models reproduced them.
The more provocative claim is about the psychology of using these tools. Howard compares AI-assisted coding to gambling — specifically to slot machines.
"The thing about AI based coding is that it's like a slot machine in that you have an illusion of control. You can craft your prompt and your list of MCPs and your skills and whatever, but in the end, you pull the lever. It's got all these hallmarks of loss disguised as a win." — [Jeremy Howard, 48:17](https://www.youtube.com/watch?v=dHBEQ-Ryo24&t=2897)
The mechanism is worth breaking down. You craft a detailed prompt. You curate your context. You feel in control. Then you submit and wait. The output either works or it does not. When it works, you feel skilled. When it fails, you re-prompt — another pull. The stochastic reinforcement schedule is identical to a slot machine: intermittent rewards, illusion of agency, and a steady drain on the human operator's time and understanding.
The result is developers who ship fragments without comprehending the system they are assembling. They can describe what the code does in isolation. They cannot explain why the architecture looks the way it does, because they did not design the architecture. The model did, and the model's design is an average of its training data — not a response to the specific constraints of the problem at hand.
Howard points to a recent Anthropic study as the strongest evidence against the productivity narrative. Anthropic's own CEO, Dario Amodei, has repeatedly claimed that AI coding tools accelerate developer output and deepen understanding. The study told a different story.
"This study from Anthropic a couple of weeks ago contradicted Dario completely because it even said that there were a few people in the study that were asking conceptual questions and they had a gradient of learning, but most people didn't." — [Jeremy Howard, 36:59](https://www.youtube.com/watch?v=dHBEQ-Ryo24&t=2219)
Most participants in the study did not gain conceptual understanding from using AI coding tools. A small subset — the ones who used the tools to ask conceptual questions rather than generate code — showed learning gains. The majority used the tools to produce output. They shipped more, understood less, and walked away with thinner mental models of their own software.
This is the core tension. The benchmark scores measure output. [SWE-bench Verified](https://benchmark.space/benchmark/swe-bench-verified) tracks whether a patch resolves an issue. It does not track whether the developer who prompted the patch could reproduce it, explain it, or modify it without the model next time. A 93.9% score on SWE-bench measures the model. It tells you nothing about the human in the loop.
Howard is not a blanket skeptic. He credits models with genuine intelligence — but only within boundaries. The [ARC-AGI](https://benchmark.space/benchmark/arc-agi-2) benchmark proved this.
"I would now say that these models can be said to be intelligent in restricted domains. The ARC challenge showed that. So if you place constraints into the problem, you can go faster towards a known goal." — [Jeremy Howard, 79:15](https://www.youtube.com/watch?v=dHBEQ-Ryo24&t=4755)
The qualification matters. "Restricted domains" and "known goal" are doing real work in that sentence. When the problem space is bounded and the objective is clear, LLMs perform well. When the task requires deciding what the objective should be — which is what software engineers actually do — the models contribute nothing that was not already in their training distribution.
| Capability | LLMs | Software engineers |
| Reproduce known patterns | Strong | Strong |
| Design novel abstractions | Weak | Core skill |
| Explain *why* not *what* | Cannot | Must |
| Operate outside training distribution | Cannot | Must |