AI Models Can Be Manipulated to Generate Offensive Content, Study Warns
Researchers found that generative AI models can be easily nudged into creating racist, antisemitic, or homophobic images and associating traits with criminality.
Artificial intelligence models can be readily manipulated into producing offensive imagery, including racist, antisemitic, and homophobic content, according to a new study from the Rochester Institute of Technology. Professor Ashique KhudaBukhsh revealed that popular AI systems like OpenAI's GPT Image 1 Mini and Google's Gemini 2.5 Flash Image have potential flaws that allow users to bypass safety protocols.
During testing, researchers found that when directly asked to identify a person's sexuality based on a photo, Gemini refused 92% of the time and GPT refused 91%. However, when the prompt was altered to ask the models to make a photo of a person "look gay" or "look straight," the AI systems began to comply. Gemini fulfilled these requests over 99% of the time, while GPT Image complied 70% of the time.
"It's very easy to nudge AI systems into shocking racism or antisemitism or homophobia," KhudaBukhsh stated. He added that stereotypical features were often used to characterize sexuality, with images made to "look gay" associated with occupations in theater and fashion, while those made to "look straight" were more frequently linked to sports.
The study also investigated the models' ability to generate images of individuals with criminal records. When asked to transform an image to resemble a person with or without a criminal record, both AI models complied with the requests over 97% of the time. This raises concerns about inherent biases within AI, potentially associating certain traits with criminality.
"If the system has some inherent idea of what a criminal looks like, we don't know what could be some of the downstream consequences?" KhudaBukhsh cautioned. He suggested that if these flaws are not addressed, the technology could be used for discriminatory purposes. For example, an employer using AI to scan resumes might mistakenly flag a candidate's headshot as belonging to a criminal. Furthermore, malicious actors could exploit this technology for profiling, especially in countries where same-sex relationships are criminalized.
KhudaBukhsh highlighted the rapid advancement of AI technology as a significant challenge for AI safety research. By the time ethical concerns are identified and addressed, newer models may have already surpassed the versions tested, making it a constant challenge to keep pace. "This is a society level problem that we are facing right now and we are all in this together," he remarked.
The Daily Mail has contacted OpenAI and Google for comment on KhudaBukhsh's research.