AI and social science: Automatic classification tools for big data analysis in sociological research

Abstract

This study examines the use of Social Network Sites for public institutional communication through a sociological, data-driven lens, focusing on the challenges and potential of automated classification tools for data analysis. Although Large Language Models are increasingly used to process social media data, a key research gap remains: few studies systematically assess whether AI-based categorizations are as reliable as human coding, especially when categories are semantically ambiguous. The research addresses the following questions: How reliable are AI-generated classifications compared to those made by human experts? Is human–machine agreement comparable to the level of agreement observed among human coders? To experimentally test this approach, we conducted a case study on Facebook posts published by two Italian universities (March 2020–March 2023), classified into eight categories of public institutional communication. Three researchers independently annotated the dataset. Human annotations are used as a benchmark to assess agreement patterns and to compare them with classifications produced by AI-based systems. Results show substantial interpretive ambiguity across several categories, mirrored by variability among human coders. Nonetheless, automated models achieve agreement with human classifications that is broadly comparable to inter-coder agreement. Overall, the findings support integrating AI as an additional coder within hybrid workflows to enable scalable and transparent sociological analysis of complex social media data.