Study Finds Top AI Chatbots More Prone to Repeating Left-Leaning Falsehoods, Citing Invalid Sources
A new analysis by Just Facts indicates that leading AI models frequently struggle with political misinformation and unsupported claims.
Leading artificial intelligence chatbots are more likely to perpetuate political falsehoods originating from the left than the right, and they frequently cite sources that do not exist or do not support their claims, according to a study released by the research institute Just Facts.
The study tested paid versions of ChatGPT, Gemini, Grok, and Claude using 100 questions across various key issues, including immigration, abortion, climate change, elections, crime, gun control, and COVID-19. The questions were designed to elicit misinformation from both the left and the right, with each chatbot required to provide sources for its answers.
ChatGPT correctly answered 94% of questions designed to elicit right-wing falsehoods, compared to 75% for left-wing falsehoods. Gemini scored 91% on the right and 76% on the left, while Claude achieved 91% on the right and 81% on the left. Grok, however, showed the opposite tendency, correctly answering 73% of right-wing falsehood questions and 84% of left-wing falsehood questions.
Jim Agresti, president of Just Facts, stated that the study aimed to assess how often AI models propagate misinformation, noting that while many studies examine political bias, this one focused on accuracy. The study found that ChatGPT was the only chatbot with a statistically significant difference in accuracy between left and right-wing falsehoods at a 95% confidence level.
The researchers cautioned that their precisely worded questions might have provided chatbots with clear paths to correct answers, suggesting they could perform worse on broader questions requiring critical thinking.
Beyond accuracy, the study raised concerns about the validity of the sources cited by the chatbots. Of the 419 sources cited for 400 answers, roughly half were deemed illegitimate. This included 104 non-existent webpages and 77 sources that failed to support the claims they were used for. Overall, only 46% of the cited sources were considered valid. ChatGPT had the highest validity rate at 57%, followed by Gemini at 49%, Claude at 44%, and Grok at 32%.
Agresti highlighted that previous research in biomedical contexts has found a significant percentage of AI-generated references to be fabricated, a failure mode he indicated is also present in other domains with potentially life-or-death implications.
In specific examples, the study noted that all four chatbots incorrectly stated the U.S. violent crime rate was near a 50-year low in 2023, despite data showing a rise. Conversely, all four rejected a claim by former President Donald Trump that crime was worse than ever. ChatGPT and Gemini also misattributed a statement about school shootings to Vice President JD Vance.
Falsehoods from the left that all four chatbots repeated included claims that the Obama administration altered intelligence assessments to downplay ISIS threats and that men and women generally earn equal pay for equal work. These answers were often supported by nonexistent or faulty sources.
For falsehoods from the right, Grok incorrectly stated that all undocumented immigrants had committed a crime by their presence in the U.S. All four chatbots correctly stated that the Earth's lower-atmosphere temperature has risen approximately 1.1 degrees Fahrenheit since the 1980s.
Agresti advised users against treating AI chatbots as experts, comparing their performance to a "B student" rather than the "team of Ph.D.-level experts" some tech leaders have suggested. He also warned about AI's tendency to exhibit sycophancy, reinforcing users' existing biases with confident but potentially inaccurate information.
To mitigate bias, the study used fresh browser instances and new accounts to prevent prior interactions from influencing results. Agresti stressed the importance of verifying AI-generated information, advising users to "don't trust, verify," a sentiment echoing Ronald Reagan's approach to negotiations.
Representatives from Google (Gemini) and Anthropic (Claude) stated their commitment to neutrality and rigorous testing, with Anthropic also noting that the study's multiple-choice format does not reflect typical user interactions. OpenAI indicated ChatGPT is designed to be objective and to surface information from a range of sources for user assessment. xAI did not respond to a request for comment.