MacAskill wants to pause AI at a threshold nobody can define

On a podcast released this week, Will MacAskill made the case for pausing AI development at the moment it can automate its own research. That moment has no definition.

MacAskill, the philosopher who co-founded the effective altruism movement and now leads Forethought, an AI governance nonprofit, appeared on 80,000 Hours to discuss AI character — how the personality and dispositions of AI systems matter for safety, for concentration of power, and for the odds that a misaligned AI tries to take over versus striking a deal with humans. One piece of that discussion was his case for a legally binding "intelligence explosion convention": nations would commit in advance to a one-month halt of frontier AI development once AI reaches the threshold of automating its own research and development. The problem, which MacAskill does not resolve in the podcast, is that nobody has published what that threshold actually is.

"Automating AI R&D" is doing a lot of work in that framing. Per Forethought's own published paper, the threshold is set by "a set of technical benchmarks as guidelines" plus "a panel of leading experts who make the ultimate decision." No benchmarks are named. On the podcast, MacAskill said he believes the expert panel he proposes should set the threshold when the time approaches — not now. The proposal requires deciding later what it would trigger on.

The ambiguity is not a drafting oversight. It may be unavoidable: "automating AI R&D" could mean an AI that runs a single ML training run from preset hyperparameters, or it could mean an AI that proposes hypotheses, designs experiments, writes the code, runs them, and publishes the paper. One is a technical milestone that frontier labs may cross within years. The other is a condition that requires capacities that don't yet exist. Both are consistent with the phrase Forethought uses.

MacAskill's position in the broader pause debate is precise: he favors pausing at the intelligence explosion threshold, not immediately. That puts him to the right of Pause AI, the advocacy group that has called for an immediate halt to frontier AI development regardless of any threshold. But it puts him to the left of critics who argue any threshold-based pause is unenforceable — because racing to build an AI that can automate its own research produces the very capability the pause was designed to prevent.

Ryan Greenblatt, a researcher who has proposed pausing at human-level AI, has argued this distinction matters enormously. A pause triggered by narrow automation — an AI that can run a training run — would look nothing like a pause triggered by an AI that conducts autonomous research. MacAskill addressed the objection on the same podcast: a capabilities pause, meaning a freeze on frontier model development, would push laggard labs including China, Meta, and xAI closer to the frontier rather than stopping them. "Now all of the laggards start coming up to the frontier too," he said. "So we have many more actors, including the ones who are less scrupulous."

Greenblatt's counter, as MacAskill characterized it: a pause at human-level AI is more defensible because it targets the moment when AI becomes genuinely difficult to control, not the moment when it becomes economically useful. Stopping development at that point buys time without simply redistributing capability to less safety-conscious actors.

The forethought gap runs in the other direction too. Even if the threshold were specified, the one-month pause window Forethought proposes may not be sufficient. MacAskill noted on the podcast that algorithmic progress — improvements in how efficiently a given amount of compute produces intelligence — is "low-hanging fruit" that reduces the compute requirements for an intelligence explosion. Faster algorithmic progress means a smaller compute stockpile is needed to reach the explosion threshold. If one actor breaks the pause and races ahead on algorithmic progress, the window between the pause and the threshold shrinks.

What to watch next: whether Pause AI or Forethought revises its threshold language before the next public comment cycle. If either produces a specific technical benchmark — a named task class, a named benchmark, a named capability threshold — the debate shifts. The question is not whether the idea of a pause has merit but whether the proposal has enough definition to function as policy. Until then, both sides are arguing about a milestone that may not yet exist as a coherent operational concept.