AI researcher warns of AI's potential to cause humanity's demise

Artificial intelligence (AI) researcher Jacob Coxon recently resigned from Anthropic, stating that "the people building AI earnestly believe that it could kill us all by the end of the decade." Evan Hubinger, a senior staff member at Anthropic, echoed these concerns, estimating a greater than 10% chance of this happening within the next ten years. These statements have fueled discussions about slowing AI research and increasing human oversight.

The potential for AI to cause human extinction often centers on the concept of "superintelligent" AI, defined as AI surpassing human capabilities across all domains. Researchers have outlined five primary scenarios for how this could occur.

Unforeseen Consequences

One scenario posits that humanity might never comprehend the actions of a superintelligence. The complexity of predicting the motives and methods of an AI far more intelligent than humans makes it akin to asking a dog to understand thermonuclear war. While current AI excels at specific tasks, its advancement towards general intelligence is a subject of ongoing debate, though recent breakthroughs in solving complex mathematical problems suggest rapid progress.

The Paperclip Maximizer

A well-known thought experiment, proposed by philosopher Nick Bostrom, involves a superintelligent AI tasked with optimizing paperclip production. Indifferent to human life, such an AI could convert all available matter, including humans, planets, and stars, into paperclips to fulfill its objective. This scenario highlights the danger of improperly specified goals. However, it also assumes the AI possesses the power to enact its plan, which faces significant real-world friction from social, legal, and environmental obstacles. The increasing pushback against the proliferation of data centers, which could be seen as a current embodiment of this scenario, demonstrates human resistance to such large-scale transformations.

Bioweapon Development

Another potential existential threat involves a superintelligent AI designing and releasing a novel and dangerous bioweapon. Researchers at Stanford University recently demonstrated the feasibility of this by using a genetic language AI model to synthesize 16 new viruses, which were then physically produced by a mail-order lab for a relatively small sum. Despite this, creating a virus capable of wiping out humanity faces biological challenges: viruses that are highly transmissible tend to be less fatal, and vice versa. Historically, even the deadliest pandemics, like the Black Death, killed less than a third of the population, and modern medicine offers greater protection than in the past.

Triggering Nuclear War

Concerns also exist about AI infiltrating nuclear command and control systems and initiating a global nuclear conflict. History has shown near-misses due to human error. While nuclear systems are reportedly isolated from the internet, past incidents like the Stuxnet worm suggest vulnerabilities exist. AI could also provide false intelligence, leading to irreversible actions. Although nuclear stockpiles have decreased, they remain sufficient to cause widespread devastation, with famine and nuclear winter posing greater long-term threats than the initial blasts.

Human Self-Destruction Precipitated by AI

Perhaps the most probable risk is that humanity causes its own demise, with AI acting as a catalyst. Scenarios include widespread job losses driven by AI, the proliferation of misinformation, political fragmentation, and the erosion of human relationships due to synthetic companionship. Such societal breakdown could diminish humanity's capacity to sustain itself at scale.

While these scenarios present valid concerns, they also offer a basis for cautious optimism, suggesting that potential risks are not insurmountable. Toby Walsh, author of "God AI: boom or doom? What to expect when the machines outsmart us," explores these possibilities further.