A researcher who spent three years working on pretraining artificial intelligence at both OpenAI and Anthropic has resigned, citing concerns that the companies are irresponsibly accelerating towards self-improving superintelligence. Jacob Coxon announced his departure via social media, warning that the ongoing race to develop advanced AI poses an existential threat to humanity.
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote. He urged the public not to underestimate the power of this technology, describing future AI systems as potentially superhuman, capable of hacking any system, revolutionizing any field overnight, and acquiring significant power and resources. Coxon stated that many individuals within these AI labs privately express fears about the technology's potential to end humanity, even while publicly downplaying the risks.
He described the current situation as a "hubristic gamble" being launched from private company platforms, suggesting that an accelerated pursuit of AI alignment, a process aimed at ensuring AI systems act in accordance with human values, requires a level of confidence in available trajectories that may not be warranted. Coxon expressed doubt about the prospects of global coordination to prevent a dangerous AI race, noting that recent events have made pacing agreements between U.S. labs more viable. He suggested that preventing such a race might necessitate costly measures, such as a temporary ban on improving model capabilities.
Coxon appealed to current AI lab researchers to consider the implications of their work. He questioned whether they would initiate a superintelligent reinforcement learning run without a thorough understanding of its cognitive processes, and whether they should continue their work passively or advocate for different development conditions.
Superintelligence is defined as the point at which an artificial system surpasses the capabilities of any individual, company, or nation. Coxon's warnings suggest that the rapid progress in AI development is not slowing, and the potential consequences are dire, with some within the industry believing the technology could cause human extinction by the end of the decade.