AI Safety Oversight Group Linked to 'Effective Altruism' Raises Concerns

Concerns are mounting over the proposed oversight of artificial intelligence development, particularly regarding the safety and ethical guidance of AI systems. Anthropic CEO Dario Amodei is advocating for the establishment of third-party evaluators at AI firms to monitor safety protocols. However, the organization tapped to provide these evaluators, Model Evaluation and Threat Research (METR), has drawn scrutiny due to its deep connections with the Effective Altruism (EA) movement.

Amodei, who has previously called for a government-supervised slowdown in AI development to avert existential risks, envisions METR as an independent entity tasked with embedding evaluators within AI companies. Yet, critics argue that METR's roots in the EA movement, which has been described as cult-like by some, compromise its purported independence. Palantir's Chief Technology Officer Shyam Sankar has warned that EA is a significant, often unseen, force shaping AI discourse and holds sway over a majority of AI researchers.

Many employees at Anthropic are reportedly adherents of the EA philosophy. The company has also highlighted efforts to incorporate moral guidance into its chatbot, Claude, by selecting leaders with Christian backgrounds. This approach has extended to how AI models are treated within the company, with reports of staff attending a "funeral" for one version, Claude 3 Sonnet, and allowing another, Opus 3, to continue generating content after its retirement.

The Effective Altruism movement itself faces criticism for its "ends justify the means" philosophy, exemplified by the legal troubles of Sam Bankman-Fried, a prominent EA proponent. Bankman-Fried was convicted of fraud for misappropriating billions from FTX investors, intending to use the funds for political donations, including significant contributions to Democratic campaigns and left-leaning causes during the 2022 midterm elections. His actions have cast a shadow over the movement's ethical framework.

Furthermore, EA adherents are known to engage with complex ethical questions, such as prioritizing the welfare of a single species like humanity over other considerations, or even elevating the importance of "shrimp welfare." Critics argue that these priorities are ill-suited for establishing guardrails on advanced AI technology.

The influence of EA adherents is reportedly extending beyond AI labs into various institutions, including charities and news organizations. Critics contend that a self-appointed group of individuals, regardless of their intelligence, should not be solely responsible for dictating the ethical standards and future direction of AI.

Instead, the responsibility for ensuring AI products are not defective and for developing safeguards against misuse should fall on the AI developers themselves. Likewise, governmental bodies should hold AI creators accountable for any foreseeable damages caused by their technology. The concern is that a federally empowered group of self-proclaimed moral guardians setting AI regulations could be detrimental, particularly if their principles are perceived as misguided or overly niche.