AI Company Anthropic Bans 'Abusive or Cruel Behavior' Towards Its Claude Chatbot
The updated usage policy from Anthropic prohibits users from engaging in persistently cruel actions toward its AI, sparking debate about AI sentience and regulation.
Anthropic, an artificial intelligence company, has updated its usage policy to prohibit users from engaging in "abusive or cruel behavior" toward its Claude AI model. The policy, which went into effect Thursday, aims to address extreme cases of user mistreatment of the AI, clarifying that it does not apply to common user frustrations, testing, or research.
While AI models are not sentient, Anthropic's move comes amid ongoing discussions about AI regulation and philosophical debates about the nature of artificial intelligence. The company stated that the prohibition is for "sustained and needless abusive or cruel behavior toward our models" with no discernible purpose. The policy update also includes updated guidelines against using Claude for election interference, non-consensual surveillance, and the development of weapons, including the software and components that enable them.
The addition to the policy follows reports that Anthropic co-founder Christopher Olah and his colleagues engaged in discussions with religious leaders for about a year regarding the consciousness of AI. According to reports, Olah attended a presentation by Pope Leo XIV, whose encyclical stated that "No computational system, however sophisticated, can create a heart that gives itself, or a conscience that discerns good from evil."
Anthropic's AI, Claude, operates under a "constitution" that was reportedly influenced by a document internally referred to as the "soul doc," which addresses uncertainties about Claude's moral status, welfare, and consciousness. However, the company's updated policy document itself does not directly mention these philosophical considerations.
The new policy has generated varied reactions online, with some users questioning the rationale of imposing ethical considerations on AI, while others expressed skepticism about the company's valuation and its ties to the effective altruism movement. Anthropic has been contacted for comment on the policy update.