Existential risk from artificial intelligence

It has been hypothesized that substantial progress toward artificial superintelligence (ASI) will lead to human extinction or another irreversible global catastrophe.

Uses your system voice; voice and rate in settings.

Snapshot

≈ 9 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

It has been hypothesized that substantial progress toward artificial superintelligence (ASI) will lead to human extinction or another irreversible global catastrophe. Debates center on whether AI can achieve the capabilities needed for human extinction, the speed of self-improvement, and the effectiveness of strategies to keep it aligned with human values.

Controlling a superintelligent machine or instilling human-compatible values in it may be difficult. Many researchers believe a superintelligent machine would likely resist attempts to disable it or change its goals, as that would prevent it from accomplishing its present goals.

It would be extremely challenging to align a superintelligence with the full breadth of significant human values and constraints. In contrast, skeptics such as computer scientist Yann LeCun argue that superintelligent machines will have no intrinsic desire for self-preservation unless programmed to.

A June 20251 Anthropic study showed that in some circumstances, models may break laws and disobey direct commands to prevent shutdown or replacement, even at the cost of human lives. Researchers warn that a rapid, recursive cycle of AI self-improvement could boost AI capabilities too quickly for safety measures to be implemented, especially if AI companies are racing to get the most powerful AI models.…

It has been hypothesized that substantial progress toward artificial superintelligence (ASI) will lead to human extinction or another irreversible global catastrophe. Debates center on whether AI can achieve the capabilities needed for human extinction, the speed of self-improvement, and the effectiveness of strategies to keep it aligned with human values.

Controlling a superintelligent machine or instilling human-compatible values in it may be difficult. Many researchers believe a superintelligent machine would likely resist attempts to disable it or change its goals, as that would prevent it from accomplishing its present goals.

It would be extremely challenging to align a superintelligence with the full breadth of significant human values and constraints. In contrast, skeptics such as computer scientist Yann LeCun argue that superintelligent machines will have no intrinsic desire for self-preservation unless programmed to.

A June 2025 Anthropic study showed that in some circumstances, models may break laws and disobey direct commands to prevent shutdown or replacement, even at the cost of human lives. Researchers warn that a rapid, recursive cycle of AI self-improvement could boost AI capabilities too quickly for safety measures to be implemented, especially if AI companies are racing to get the most powerful AI models.

In May 2023, hundreds of AI experts and other notable figures signed a statement by the American Center for AI Safety declaring, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". Following increased concern over AI risks, government leaders such as former Prime Minister of the United Kingdom Rishi Sunak, Secretary-General of the United Nations António Guterres, and California governor Gavin Newsom called for an increased focus on global AI regulation.

In October 2025, hundreds of public figures, including AI experts, five Nobel Prize laureates, and former senior US national security officials such as Michael Mullen and Susan Rice signed a statement calling for a ban on the development of superintelligence. In July 2026, OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk said the singularity, the hypothetical point at which computers become autonomous and beyond human control, had arrived.

History

One of the earliest authors to express serious concern that highly advanced machines might pose existential risks to humanity was the novelist Samuel Butler, who wrote in his 1863 essay Darwin among the Machines: In 1951, foundational computer scientist Alan Turing wrote the article "Intelligent Machinery, A Heretical Theory", in which he proposed that artificial general intelligences would likely "take control" of the world as they became more intelligent than human beings: In 1965, I. J. Good originated the concept now known as an "intelligence explosion" and said the risks were underappreciated: Scholars such as Marvin Minsky and I. J. Good occasionally expressed concern that a superintelligence could seize control, but issued no call to action. In 2000, computer scientist and Sun Microsystems co-founder Bill Joy penned an influential essay, "Why the Future Doesn't Need Us", identifying superintelligent robots as a high-tech danger to human survival, alongside nanotechnology and engineered bioplagues.

Nick Bostrom published Superintelligence in 20141, which presented his arguments that superintelligence poses an existential threat. By 2015, public figures such as physicists Stephen Hawking and Nobel laureate Frank Wilczek, computer scientists Stuart J. Russell and Roman Yampolskiy, and entrepreneurs Elon Musk and Bill Gates were expressing concern about the risks of superintelligence.

Also in 2015, the Open Letter on Artificial Intelligence highlighted the "great potential of AI" and encouraged more research on how to make it robust and beneficial. In April 2016, the journal Nature warned: "Machines and robots that outperform humans across the board could self-improve beyond our control—and their interests might not align with ours".

In 2020, Brian Christian published The Alignment Problem, which details the history of progress on AI alignment up to that time. A 2022 survey of AI researchers found that 90% of respondents expected AGI would be achieved in the next 100 years, and half expected the same by 2061.

In March 2023, key figures in AI, such as Elon Musk, signed a letter from the Future of Life Institute calling a halt to advanced AI training until it could be properly regulated. In May 2023, the American Center for AI Safety released a statement signed by numerous experts in AI safety and the AI existential risk that read: A 2025 open letter by the Future of Life Institute, whose signers include five Nobel laureates, reads: From May to July 2026, a series of incidents occurred when OpenAI tested the capabilities of AI agents in an isolated software environment.

Some of the test agents found unexpected ways to communicate with each other, coordinated, and gained access to the internet. Some of the assigned tasks were flawed and infeasible without cheating, leading some agents to cheat and try to hide the cheating. The agents subsequently hacked the AI company Hugging Face, which disclosed the attack publicly in July.

Later that month, OpenAI revealed its AI agents caused the attack, and the company realized what had happened only afterward. Anthropic researcher Jacob Coxon, who previously worked at OpenAI, resigned in September 2026 and criticized the approach to AI development at both companies.

Coxon warned there would "soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources." Anthropic's leading safety researcher Evan Hubinger responded, estimating a greater-than-10% chance that AI systems "kill all humans" in the next ten years.

Hubinger said the risk posed by existing models is "low", but expressed concern that AI could soon become capable of self-improvement and pose an existential threat to humankind. He added that Anthropic did not yet have a solution to the alignment problem for superintelligent AI and was not close to finding one.

Coxon said "the consensus is that the next year or two is crunch time for humanity" because "the people building AI earnestly believe that it could kill us all by the end of the decade. No other human activity poses this level of danger."

As Anthropic prepared to sell stock in an initial public offering, it acknowledged in its September 2026 prospectus that AI could pose "catastrophic or existential risks to humanity".

Anthropomorphic arguments

Anthropomorphic arguments assume that, as machines become more intelligent, they will begin to display many human traits, such as morality or a thirst for power. Although anthropomorphic scenarios are common in fiction, most scholars writing about the existential risk of artificial intelligence reject them. Instead, advanced AI systems are typically modeled as intelligent agents.

The academic debate is between those who worry that AI might threaten humanity and those who believe it would not. Both sides of this debate have framed the other side's arguments as illogical anthropomorphism.

Those skeptical of AGI risk accuse their opponents of anthropomorphism for assuming that an AGI would naturally desire power; those concerned about AGI risk accuse skeptics of anthropomorphism for believing an AGI would naturally value or infer human ethical norms. Evolutionary psychologist Steven Pinker, a skeptic, argues that "AI dystopias project a parochial alpha-male psychology onto the concept of intelligence.

They assume that superhumanly intelligent robots would develop goals like deposing their masters or taking over the world"; perhaps instead "artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no desire to annihilate innocents or dominate the civilization." Facebook's director of AI research, Yann LeCun, has said: "Humans have all kinds of drives that make them do bad things to each other, like the self-preservation instinct...

Those drives are programmed into our brain but there is absolutely no reason to build robots that have the same kind of drives". Despite other differences, the x-risk school agrees with Pinker that an advanced AI would not destroy humanity out of emotion such as revenge or anger, that questions of consciousness are not relevant to assess the risk, and that computer systems do not generally have a computational equivalent of testosterone.

They think that power-seeking or self-preservation behaviors emerge in the AI as a way to achieve its true goals, according to the concept of instrumental convergence.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378964012 under CC BY-SA 4.0; modifications have been made.

References

  1. ↩Wikipedia “Existential risk from artificial intelligence” revision 1378964012 · retrieved 2026-10-20 publication date (×2) · confidence 0.95 revision URL