AI in Policing Risks Exacerbating Discrimination, Researchers Warn

The increasing integration of artificial intelligence in law enforcement carries a significant risk of amplifying discrimination against minority groups, according to researchers. Studies indicate that AI systems used to predict crime hotspots may rely on biased data, perpetuating and even intensifying existing prejudices through a harmful feedback loop.

Experts are advocating for a comprehensive review of AI in policing, emphasizing the need for enhanced human oversight. Dr. Carolyn McKay, an associate professor and researcher in digital justice at the University of Sydney Law School, suggests that law enforcement agencies must meticulously examine the data used to identify and eliminate inherent biases. "The public functions of state policing ought to be subject to human oversight and scrutiny," McKay stated.

McKay likens the issue to an AI system trained on outdated maps. If an AI is tasked with predicting crime locations based on historical data that already reflects human biases, it can create self-fulfilling prophecies. "The problem is as old as policing, but AI may be speeding it up," she noted.

Sociologist Sarah Brayne of the University of Texas at Austin, who studied the Los Angeles Police Department's use of big data, found that the force increasingly utilized data for crime prediction rather than solely for response. Her research, published in the American Sociological Review, revealed that the data fed into these systems was skewed, with crimes in public spaces being more readily recorded and police attention disproportionately focused on Black neighborhoods. Brayne observed that once this biased data is processed, "the predictions appear impartial," creating a feedback loop where the AI's predictions contribute to the very occurrences they forecast.

However, Dr. Dimitrios Kollias, an associate professor in multimodal artificial intelligence at Queen Mary University of London, argues that modern AI technology has advanced to account for historical data biases. "Modern AI systems do not simply retrieve or copy their training examples; they can combine learned information in new ways and generate outputs and solutions that were not explicitly present in those examples," Kollias explained. He cautioned, though, that as AI becomes more autonomous, ensuring its behavior remains safe, reliable, and under meaningful human control is crucial.

A significant challenge lies in addressing the often-flawed information stored by authorities that AI systems draw upon. Researchers at New York University highlighted the issue of "dirty data"—police records shaped by potentially corrupt, biased, or unlawful practices, including manipulated crime figures and arrests of innocent individuals. Their study found that in a majority of US areas examined, police data from periods of biased policing was available for training AI systems. The AI, unable to discern data flaws, perpetuates these patterns, leading to predictions that reflect past biases rather than objective crime distribution.

McKay proposes that independent audits of data used for AI mapping could help correct erroneous lines. However, rectifying this "dirty data" is complex, as mistakes are deeply embedded. Technical solutions offered by AI vendors may be insufficient without independent verification of the data's origins and collection methods.

AI's reliance on provided data can lead to significant errors. An example from US healthcare, detailed in a 2019 Science study, illustrated how an AI system designed to predict patient healthcare costs, and thus allocate support, exhibited bias. Because less money was spent on Black patients than on equally ill white patients, the AI, by treating higher costs as an indicator of greater need, unfairly favored white patients. When the system was adjusted to predict patient illness severity instead of cost, the bias diminished, significantly increasing the proportion of Black patients identified for extra help.

Kollias acknowledged that AI bias from flawed data is a "an important concern" but asserted that "this does not mean that bias is inevitable." He suggested methods like testing systems for unfair outcomes, improving data and objectives, and implementing safeguards. He also proposed providing AI with contextual data, such as areas of neglect or underfunding, to offer a more holistic understanding, potentially leading to community-based interventions rather than solely increased policing.

Despite efforts to focus AI on environmental factors rather than individuals, concerns persist that such systems can still lead to over-policing. McKay pointed out that even when AI targets locations, the inhabitants and their circumstances remain linked to historical neglect and heavy policing, meaning changes in the target may not resolve the underlying issue.

Ultimately, human bias is the root cause. AI, devoid of its own beliefs, merely reflects the biases present in the data it receives. Kollias suggested that well-designed, independently evaluated AI systems could, paradoxically, make these patterns more measurable and visible, offering an opportunity to identify and reduce them. However, he stressed the necessity of human involvement, drawing parallels to the healthcare study where choices were checked and altered. "Otherwise, like a map-maker inking over old pencil lines, AI risks leaving us all in a windowless room, permanently trapped in yesterday's mistakes," the researchers concluded.