Measuring criticism of the police in the local news media using large language models

Abstract

High-profile incidents of police violence against Black citizens over the past decade have spawned contentious debates in the United States on the role of police. This debate has played out prominently in the news media, leading to a perception that media outlets have become more critical of the police. There is currently, however, little empirical evidence supporting this perceived shift. We construct a large dataset of local news reporting on the police from 2013 to 2023 in 10 politically diverse U.S. cities. Leveraging advanced language models, we measure criticism by analyzing whether reporting supports or is critical of two contentions: 1) that the police protect citizens and 2) that the police are racist. To validate this approach, we collect labels from members of different political parties. We find that contrary to public perceptions, local media criticism of the police has remained relatively stable along these two dimensions over the past decade. While criticism spiked in the aftermath of high-profile police killings, such as George Floyd’s murder, these events did not produce sustained increases in negative police news. In fact, reporting supportive of police effectiveness has increased slightly since Floyd’s death. We find only small differences in coverage trends in more conservative and more liberal cities, undermining the idea that local outlets cater to the politics of their audiences. Last, although Republicans are more likely to view a piece of news as supportive of the police than Democrats, readers across parties see reporting as no more critical than it was a decade ago.