AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science
Abstract
Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.
Article Details
Journal Info
Proceedings of the National Academy of Sciences
National Academy of Sciences
Authors (272)
Abel Brodeur
David Valenta
Department of Economics, Faculty of Social Sciences, University of Ottawa
Alexandru Marcoci
Institute for Technology and Humanity, University of Cambridge
Juan P. Aparicio
Department of Economics, Faculty of Social Sciences, University of Ottawa
Derek Mikola
Bruno Barbarioli
Department of Economics, Faculty of Social Sciences, University of Ottawa
Rohan Alexander
Department of Statistical Sciences, Faculty of Information, University of Toronto
Lachlan Deer
Tom Stafford
Lars Vilhuber
Gunther Bensch
Climate and Development Policy Division, Leibniz Institute for Economic Research - Leibniz Institute for Economic Research
Fabio Motoki
Robert C. Vackar College of Business and Entrepreneurship, University of Texas Rio Grande Valley
Mohamed Abdelhady
Department of Economics, Carleton University
Yousra Abdelmoula
Statistics Canada
Ghina Abdul Baki
Department of Economics, Faculty of Social Sciences, University of Ottawa
Tomás Aguirre
GovAI
Sriraj Aiyer
Department of Experimental Psychology, University of Oxford
Shumi Akhtar
Finance Discipline, University of Sydney Business School, The University of Sydney
Farida Akhtar
Department of Actuarial Studies and Business Analytics, Macquarie Business School, Macquarie University
Melle R. Albada
Department of Socioeconomics, Vienna University of Economics and Business
Micah Altman
David Angenendt
Technical University of Munich School of Management, Technical University of Munich
Zahra Arjmandi Lari
Independent researcher
Jorge Armando De León Tejada
Facultad de Economía, Universidad del Rosario
David Rodriguez Arana
School of Economics, Universidad del Rosario
Igor Asanov
International Center for Higher Education Research and Faculty of Economics, University of Kassel
Anastasiya-Mariya Noha
International Center for Higher Education Research and Faculty of Economics, University of Kassel
Rebecca Ashong
Department of Economics, University of Ghana
Tobias Auer
Department of Economics, University of Basel
Francisco J. Bahamonde-Birke
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University
Bradley J. Baker
Söhnke M. Bartram
Warwick Business School, University of Warwick
Dongqi Bao
Department of Economics, University of Zurich
Lucija Batinovic
Department of Behavioural Sciences and Learning, Linköping University
Tommaso Batistoni
Centre for Experimental Social Sciences, Nuffield College, University of Oxford
Monica Beeder
Louis-Philippe Beland
Carsten Gero Bienz
Department of Finance, Norwegian School of Economics
Christ Billy Aryanto
Faculty of Psychology, Atma Jaya Catholic University of Indonesia
Cylcia Bolibaugh
Department of Education, University of York
Carl Bonander
Ramiro Bravo
Faculty of Biology Medicine and Health, The University of Manchester
Egor Bronnikov
School of Business and Economics, Maastricht University
Stephan Bruns
Nino Buliskeria
Sara Caicedo-Silva
Facultad de Economía, Universidad de los Andes
Andrea Calef
Juan Sebastian Cano Arias
School of Economics, Universidad del Rosario
Gustavo A. Castillo Alvarez
Department of Economics, Universidad de Los Andes
Solomon Caulker
United Methodist University Sierra Leone
Simonas Cepenas
Department of Economics, ISM University of Management and Economics
Arthur Chatton
Département de Médecine Sociale et Préventive, Université Laval
Zirou Chen
Department of Management, Marketing and Information Systems, Hong Kong Baptist University
Ngozi Chioma Ewurum
Department of Economics, College of Management Sciences, Michael Okpara University of Agriculture
Anda-Bianca Ciocîrlan
School of Psychology, University of Sheffield
Felix J. Clouth
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University
Jason Collins
Nikolai Cook
Cesar Cornejo
Department of Medical Statistics, The London School of Hygiene & Tropical Medicine
João Craveiro
School of Psychology, University of Sheffield
Jonathan Crechet
Jing Cui
Niveditha Chalil Vayalabron
School of Earth and Planetary Science, National Institute of Science Education and Research
Christian Czymara
Carlos Daniel Bermúdez Jaramillo
Department of Economics, Universidad del Rosario
Hannes Datta
Department of Marketing, Tilburg University
Lien Denoo
Department of Strategy & Entrepreneurship, Tilburg University
Arshia Dhaliwal
Department of Economics, Carleton University
Nency Dhameja
Department of Economics, Binghamton University
Elodie Djemai
Economics, Université Paris Dauphine-Paris Sciences et Lettres
Erwan Dujeancourt
Swedish Institute for Social Research, Stockholm University
Uǧurcan Dündar
Department of Marketing, Vienna University of Economics and Business Vienna
Thibaut Duprey
Yasmine Eissa
Youssef El Fassi
Department of Accounting and Control, Hautes études commerciales Lausanne
Ismail El Fassi
Institute for Accounting, Controlling and Auditing, University of St. Gallen
Keaton Ellis
Monash University
Ali Elminejad
Mahmoud Elsherif
School of Psychology and Vision Science, University of Leicester
Aysil Emirmahmutoglu
Department of Business and Management Science, Norwegian School of Economics
Giulian Etingin-Frati
Emeka Eze
Department of Economics, College of Management Sciences, Michael Okpara University of Agriculture
Jan Fabian Dollbaum
Jan Feld
Andres Felipe Rengifo Jaramillo
Business School, Universidad de los Andes
Guidon Fenig
Victoria Fernandes
Vancouver School of Economics, University of British Columbia
Lenka Fiala
Lukas Fink
Department of Economics, School of Business and Economics, Freie Universität Berlin
Mojtaba Firouzjaeiangalougah
Sara Fish
Jack Fitzgerald
Department of Ethics, Governance, and Society, School of Business and Economics, Vrije Universiteit Amsterdam
Rachel Forshaw
Department of Accountancy, Economics and Finance, Heriot-Watt University
Alexandre Fortier-Chouinard
Louis Fréget
laboratoire d’Économie de Dauphine, Centre Pour la Recherche EconoMique et ses Applications
Joris Frese
Jacopo Gabani
Sebastian Gallegos
Max C. Gamill
School of Chemical, Materials and Biological Engineering, University of Sheffield
Attila Gáspár
Romain Gauriot
Evelina Gavrilova
Diogo Geraldes
Giulio Giacomo Cantone
Grant Gibson
Dirk Goldschmitt
School of Psychology, University of Sheffield
Amélie Gourdon-Kanhukamwe
Andrea Gregor de Varda
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
Idaliya Grigoryeva
Alexi Gugushvili
Aaron H. A. Fletcher
School of Computer Science, University of Sheffield
Florian Habermann
University College Dublin Lochlann Quinn School of Business, University College Dublin
Márton Hablicsek
Mathematics Institute, Leiden University
Joanne Haddad
Jonathan D. Hall
Olle Hammar
Malek Hassouneh
Department of Economics, University of Toronto
Carina I. Hausladen
Sophie C. F. Hendrikse
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University
Matthew Hepplewhite
Anson T. Y. Ho
Senan Hogan-Hennessy
Department of Economics, Cornell University
Elliot Howley
School of Psychology, University of Nottingham
Gaoyang Huang
Department of Economics, University of Zurich
Héloïse Hulstaert
Centre for Environmental Sciences, Hasselt University
Zlatomira G. Ilchovska
School of Psychology, University of Nottingham
Paola Jaimes Santamaria
Political Economy Research Institute, University of Massachusetts Amherst
Niklas Jakobsson
Joakim Jansson
Department of Economics and Statistics, Linnaeus University
Ewa Jarosz
Faculty of Economic Sciences, University of Warsaw
Hossein Jebeli
Financial Stability Department, Climate Analysis Team, Bank of Canada
Yanchen Jiang
School of Engineering and Applied Sciences, Harvard University
Hiba Junaid
Bart’s Life Sciences, Bart’s Health National Health Services Trust
Rohan Kalluraya
School of Electrical and Computer Engineering, Cornell University
Sunny Karim
Edmund Kelly
Eva Kimel
Department of Psychology, University of York
Sorravich Kingsuwankul
Tinbergen Institute
Valentin Klotzbücher
Department of Economics, University of Freiburg
Daniel Krähmer
Department of Sociology, Ludwig Maximilian University of Munich
Pijus Krūminas
Department of Economics, ISM University of Management and Economics
Nicholas Kruus
Department of Politics and International Relations, University of Oxford
Essi Kujansuu
Department of Economics, Faculty of Economics and Statistics, University of Innsbruck
Christoph F. Kurz
Department of Health Economics, Ludwig-Maximilians-Universität Munich
Stephan Küster
School of Business and Economics, Freie Universität Berlin
Blake Lee-Whiting
Felix Lewandowski
School of Psychology, University of Nottingham
Tongzhe Li
Ruoxi Li
Department of Economics, Yale University
Dan Liu
Jiacheng Liu
Paul G. Allen Center for Computer Science and Engineering
Helix Lo
Department of Advanced Social and International Studies, University of Tokyo
Katharina Loter
Felipe Macedo Dias
Dyson School of Applied Economics and Management, Cornell University
Christopher R. Madan
Nicolas Mäder
Marco Mandas
Cesar Mantilla
Department of Economics, School of Administrative and Economic Sciences, Pontificia Universidad Javeriana
Jan Marcus
Diego Marino Fages
Xavier Martin
School of Economics and Management, Tilburg University
Ryan McWay
Daniel Medina-Gaspar
Department of Economics, School of Finance, Economics and Government, Universidad Escuela Finanzas Economía y Gobierno
Sisi Meng
Samuel Curtis Johnson School of Business, Cornell University
Lingyu Meng
Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, Peking University
Simon Merz
School of Economics and Business, University of Halle
Alex P. Miller
Thibault Mirabel
Equalis Capital
Dibya Deepta Mishra
Department of Economics, Rice University
Sumit Mishra
Belay W. Moges
Department of Psychology, Dilla University
Morteza Mohandes Mojarrad
Center PhD Students, Research Group: Econometrics, Tilburg School of Economics and Management, Tilburg university
Myra Mohnen
Louis-Philippe Morin
Lucija Muehlenbachs
Gastón Mullin
Research Group: Econometrics, Tilburg School of Economics and Management, Tilburg University
Andreea Musulan
Sara Muzzì
Department of Medicine and Surgery, University of Milano-Bicocca
James A. C. Myers
Department of Health Sciences, University of York
Florian Neubauer
Tuan Nguyen
Ali Niazi
Ardyn Nordstrom
Bartłomiej Nowak
Institute of Psychology, Cardinal Stefan Wyszynski University
Daneal O’Habib
Currency Department, Bank of Canada
Tim Ölkers
Justin Ong
School of Psychology, University of Sheffield
Valeria Orozco Castiblanco
Instituto de Estudios Superiores de la Empresa Business School, University of Navarra
Ömer Özak
Ali I. Ozkes
SKEMA Business School, Groupe de Recherche en Droit, Économie et Gestion, Université Côte d’Azur
Mikael Paaso
Department of Finance, Rotterdam School of Management, Erasmus University Rotterdam
Shubham Pandey
Varvara Papazoglou
School of Computer Science, University of Sheffield
Romeo Penheiro
Linh Pham
Department of Economics, Business and Finance, Lake Forest College
Ulrike Phieler
Department of Marketing, Vienna University of Economics and Business
Peter Pütz
Department of Economics, Bielefeld University
Quan Qi
Department of Economics, University at Albany, State University of New York
Jingyi Qiu
Manuel T. Rein
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University
David A. Reinstein
The Unjournal
Juuso Repo
Finnish Flagship Inequalities, Interventions, and New Welfare State Flagship Research Centre, University of Turku
Nicolas Rudolf
Department of Accounting and Control, University of Lausanne
Shree Saha
Applied Economics and Management, Cornell University
Orkun Saka
Department of Economics, City St George’s, University of London
Chiara Saponaro
Department of Psychology, University of Milano-Bicocca
Georg Sator
Martijn Schoenmakers
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University
Raffaello Seri
Meet Shah
Department of Real Estate Management, Ted Rogers School of Management, Toronto Metropolitan University
Paul Sibille
Management School, HEC Liège, University of Liege
Christoph Siemroth
Vladimir Skavysh
Data and Digital Services Department, Bank of Canada
Ben Slater
Leverhulme Centre for the Future of Intelligence, University of Cambridge
Wenting Song
Department of Economics, University of California
Stefan Staubli
Tobias Steindl
Institute for Business Administration, University of Regensburg
Nomwendé Steven Waongo
Department of Economics, Faculty of Social Sciences, University of Ottawa
Paul Stott
Department of Linguistics, Ghent University
Stephenson Strobel
Roshini Sudhaharan
Department of Marketing, Tilburg School of Economics and Management, Tilburg University
Pu Sun
Scott D. Swain
Oleksandr Talavera
Department of Economics, University of Birmingham
Hanz M. Tantiangco
School of Information, Journalism and Communication, University of Sheffield
Georgy Tarasenko
Department of Government, Cornell University
Boyd Tarlinton
Agri-Science Queensland, Department of Primary Industries, Brisbane
Mariam Tarraf
Department of Economics, Carleton University
Ken Teoh
Asia and Pacific Department, International Monetary Fund
Rémi Thériault
Bethan Thompson
School of Natural and Social Sciences, Scotland’s Rural College
Tonghui Tian
Department of Economics, Carleton University
Wenjie Tian
Emmanuel Tolani
Chair of Agricultural Production and Resource Economics, Technical University of Munich
Nicolai Borgen
Solveig Topstad Borgen
Javier Torralba
Department of Marketing, Tilburg University
Carolina Velez-Ospina
World Bank
Man Wai Mak
Department of Economics, Carleton University
Lukas Wallrich
Birkbeck Business School, Birkbeck, University of London
Zeyang Wang
Department of Economics, Vanderbilt University
Leah Ward
Division of Psychology & Mental Health, School of Health Sciences, University of Manchester
Matthew D. Webb
Duncan Webb
Center for Health and Wellbeing, Princeton University
Bryan S. Weber
Christoph Weber
Department Economics, Law, and Society, Ecole upérieure des sciences commerciales School of Management
Wei-Chien Weng
Christian Westheide
Tom Wilkinson
School of Chemical, Materials and Biological Engineering, University of Sheffield
Kwong-Yu Wong
Department of Economics, National University of Singapore
Marcin Wroński
Zhuangchen Wu
Department of Economics, University of Birmingham
Qixia Wu
Department of Economics, Faculty of Social Sciences, University of Ottawa
Victor Y. Wu
Department of Political Science, Stanford University
Bohan Xiao
Department of Economics, Faculty of Social Sciences, University of Ottawa
Feihong Xu
Department of Engineering Sciences & Applied Mathematics, Northwestern University
Cong Xu
Department of Statistics and Data Science, College of Science
Pranav Yadav
Department of Accounting, Tilburg School of Economics and Management, Tilburg University
Yu Yang Chou
Institute for Psychiatry, Psychology & Neuroscience, King’s College London
Luther Yap
Myra Yazbeck
Bo Yao
Zuzanna Zagrodzka
School of Biosciences, University of Sheffield
Tahreen Zahra
Mirela Zaneva
Xiaomeng Zhang
State Key Laboratory of Advanced Environmental Technology, Institute of Urban Environment
Ziwei Zhao
Han Zhong
Department of Marketing, Rotman School of Management, University of Toronto
Aras Zirgulis
Department of Economics, ISM University of Management and Economics
Jiacheng Zou
Department of Industrial Engineering and Operations Research, Columbia University
Floris Zoutman
Department of Business and Management Science, Norwegian School of Economics
Christelle Zozoungbo
Department of Economics, Penn State University