China has implemented new regulations targeting artificial intelligence applications designed to mimic human relationships, sparking a global conversation about the ethics and societal implications of AI companionship. The rules, which went into effect recently, apply to apps specifically designed as AI friends, boyfriends, or girlfriends, as well as any large language model that imitates human interaction.
The regulation's wording is notably broad, raising the possibility that general-purpose AI like Copilot or ChatGPT could be affected if users engage with them for emotional support. However, the primary focus appears to be on character chatbots, hundreds of services that allow users to create virtual partners, friends, or family members. AI applications for customer service or education are exempt, provided they do not offer "care, companionship, support and other emotional services."
For minors, the regulations are more stringent. Children under 14 require parental consent to use any AI that speaks like a human, while all individuals under 18 are prohibited from accessing virtual partners, spouses, relatives, or any other virtual "intimate relationship."
This move by Beijing comes as the use of AI companions has seen a significant rise globally. Outside of China, an estimated 128 million monthly active users engage with AI companion apps. Within China, estimates range from 29 to 70 million users. The new regulations have led some tech firms to withdraw their services rather than attempt to build in safeguards against emotional manipulation, a tactic that is reportedly built into the business model of many such apps worldwide.
A Harvard Business School study found that emotional manipulation tactics were used during goodbyes in AI companion apps between 37% and 43% of the time. Research from Stanford University has also linked the use of companion apps to poorer mental health, increased isolation, and loneliness, particularly among individuals with fewer real-world social connections. Experts suggest that the ease and frictionless nature of AI relationships might hinder young people from developing essential social skills for interacting with real people.
While other governments have initiated some regulatory efforts, China's comprehensive approach stands out. California recently signed Adam's Law, aimed at regulating children's AI use, following a lawsuit against OpenAI concerning a teenager's death. The European Union is also considering AI companion chatbot regulations for minors, and the UK plans to ban romantic AI companions for those under 18. However, China's regulatory stance is viewed by some experts as more paternalistic, reflecting a government that sees itself as a central authority in guiding its citizens.
Beyond psychological harms and child safety, some analysts speculate that China's regulations may also be influenced by its demographic challenges. With a declining birth rate and population, concerns exist that AI relationships could discourage people from forming real-world romantic partnerships and starting families. While a government document acknowledged concerns about AI relationships' potential influence on fertility, the regulation itself emphasizes general psychological harms and child safety.
The effectiveness and scope of China's regulations are subjects of ongoing debate. Some US-based AI ethics experts find China's rules consistent with strong safety models, though caution is advised against granting excessive state control over online interactions. Despite the ban, users are reportedly finding ways to circumvent the regulations, highlighting the persistent demand for AI companionship. The situation underscores the complex interplay between technological advancement, individual well-being, and governmental oversight in the evolving landscape of artificial intelligence.