Independent essays and ideasAboutContactDeutsch
science

AI existential risk: why the builders of superintelligence remain silent on the probability of catastrophe

Protesters have staged hunger strikes outside Anthropic in San Francisco and Google DeepMind in London demanding a halt to frontier AI research. A new book by Machine Intelligence Research Institute leaders argues for a global treaty banning superintelligent AI development, enforced by military means if necessary. While risk estimates vary from 1 percent to near certainty, the CEOs building these systems, Dario Amodei, Demis Hassabis and Sam Altman, have not publicly detailed why they believe the gamble is worth taking.

Protester on hunger strike outside a modern glass office building with Anthropic signage in San Francisco

A hunger strike outside Anthropic offices in San Francisco has entered its third week. A similar protest outside Google DeepMind in London ended quickly after medical warnings. The demonstrators share a single demand: stop the race toward superintelligent artificial intelligence, which they say threatens to destroy life on Earth.

The argument for a global ban

Eliezer Yudkowsky and Nate Soares, founder and president of the Machine Intelligence Research Institute (MIRI), have published If Anyone Builds It, Everyone Dies. MIRI, formed around 2002, was likely the first organised group dedicated to preventing AI catastrophe. Their proposal is sweeping: a global treaty modelled on nuclear non-proliferation agreements that bans all frontier AI research, backed by the threat of military strikes on violating data centres.

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

Yudkowsky and Soares structure the risk in four parts. First, a superintelligent AI would surpass human power as humans surpass chimpanzees. Second, engineers cannot inspect the trillions of connections inside a trained model any more than biologists can read a full phenotype from DNA. Third, training objectives may produce alien goals unconnected to human intent, much as evolution produced contraception alongside the drive to reproduce. Fourth, there is no safe test flight: a superintelligence must work correctly on the first activation, yet rocket history shows roughly 30 percent of new designs fail on maiden launch.

From fringe to front page

These views were once confined to specialised forums. Elon Musk, who heads the AI lab xAI, has repeatedly compared AI danger to nuclear weapons and warned of "civilisation destruction". Donald Trump, during his 2025 UK state visit accompanied by Silicon Valley chiefs including Sam Altman of OpenAI, ordered strikes on Iranian nuclear facilities earlier this year. The logical extension, whether a president would target allied data centres, is no longer purely hypothetical.

A spectrum of estimated probabilities

Yudkowsky and Soares sit at the extreme end of a contested field. The writer Scott Alexander describes himself as a "boring moderate" with a sub-25 percent doom estimate. AI researchers Quintin Pope and Nora Belrose, self-described optimists, place catastrophic takeover risk at roughly 1 percent, arguing that training resembles human learning more than evolution and is therefore easier to control. Even at the optimistic end, the numbers imply a game of Russian roulette with a 100-chamber revolver.

The silence from the builders

Dario Amodei, chief executive of Anthropic, Demis Hassabis, chief executive of Google DeepMind, and Altman all understand the arguments. Hassabis met DeepMind co-founder Shane Legg at a MIRI conference. Altman has credited Yudkowsky with influencing OpenAI's creation. Anthropic was founded with safety goals rooted in this research. Each has acknowledged existential risk in public statements. Hassabis has spoken of "incredible things for humanity" alongside existential dangers if mismanaged. Amodei authored a seminal 2016 paper on AI risks and has written extensively on potential upsides such as curing disease and ending climate change.

Yet none has published a chapter-by-chapter rebuttal of the doom case. The book the field awaits, I'm Going to Build It, And I Don't Think We're Going to Die, does not exist. The hunger striker in San Francisco, the aborted protest in London, and the widening circle of researchers assigning non-trivial extinction probabilities are left waiting for the architects of the technology to explain, in detail, why they believe the gamble is justified.

What happens next

The European Union has moved ahead with the AI Act, the world's first comprehensive horizontal regulation of artificial intelligence, which entered force in August 2024 and classifies systems by risk level. The Act does not ban frontier research but imposes transparency and evaluation obligations on the most powerful models. Meanwhile, the UK hosted the first global AI Safety Summit at Bletchley Park in 2023 and established the AI Safety Institute. International dialogue continues through the G7 Hiroshima Process and a planned summit in France. Whether voluntary commitments and regulatory frameworks can match the enforcement regime Yudkowsky and Soares envision, or whether the builders will finally articulate their safety case in full, remains the central question for a technology that its creators say could be the last invention humanity ever needs to make.