TikTok Restricted Mode Fails to Block Graphic Content, Investigation Finds
Loopholes allow AI-generated cartoons depicting violence and torture to reach underage users, with a startup facilitating their creation.
A safety setting on TikTok, intended to shield children from harmful material, is reportedly failing to prevent them from viewing graphic videos that include depictions of rape and torture. Loopholes in the platform's rules allow animated content, which does not fall under bans on 'realistic violence' or 'sexually suggestive content,' to bypass filters.
A startup firm called Director AI has developed video-creation tools that have enabled users to produce over 15,000 clips. These videos are algorithmically designed to be addictive, and some have garnered tens of millions of views. Examples include a cartoon depicting a surgical procedure to create a celebrity lookalike and a scene where a burglar assaults and robs a wealthy woman. One video in particular, showing a mother electrocuting her baby in a bathtub, was highlighted as particularly disturbing.
Director AI's co-founder, Ben Chiang, has referred to the output as 'AI slop' and boasted of billions of views for videos created using their tools. A companion app, Reel AI, offers some of these videos, and its Google Play rating suggests it is suitable for children as young as seven. Following inquiries, the age rating for Reel AI was reportedly changed.
In response to the findings, the UK's Shadow Education Secretary, Laura Trott, stated that social media companies should not have access to children until their platforms are proven safe. The Department for Culture, Media and Sport affirmed its backing for broadcast regulator Ofcom to ensure tech companies comply with the Online Safety Act, emphasizing their legal duty to remove illegal content and protect children.
An investigation by The Mail on Sunday established a TikTok account with 'restricted mode' enabled and an age set to 14. Despite limitations applied to profanity, firearms, sexually suggestive content, drugs, and violent imagery, the account's feed was inundated with content violating these restrictions. While TikTok stated that videos violating community guidelines are removed and that the specific videos cited in the report were taken down, the accounts that produced them reportedly remain active and continue to feature inappropriate material.
Separately, a report from December 2022, revealed in court documents related to a lawsuit against Meta, indicated that Instagram also faced internal findings of 'high-risk' child safety issues. On sampled days, millions of accounts flagged as potential child-safety risks were recommended to teenagers, and hundreds of thousands of potential 'violators' were shown to unconnected teenagers. Meta's then-president of global affairs, Sir Nick Clegg, reportedly forwarded the audit to Meta's legal team but was unaware of subsequent actions.