Estimating the extent and sources of model uncertainty in political science

Abstract

Assessing model uncertainty is crucial to quantitative political science. Yet, most available sensitivity analyses focus only on a few modeling choices, most notably the covariate space, while neglecting to jointly consider several equally important modeling choices simultaneously. In this article, we combine the exhaustive and systematic method of the Extreme Bounds Analysis with the more multidimensional logic underpinning the multiverse approach to develop an approach to sensitivity analyses. This allows us to systematically assess the degree and sources of model uncertainty across multiple dimensions, including the control set, fixed effect structures, SE types, sample selection, and dependent variable operationalization. We then apply this method to four prominent topics in political science: democratization, institutional trust, public good provision, and welfare state generosity. Results from over 3.6 bn estimates reveal widespread model uncertainty, not just in terms of the statistical significance of the effects, but also their direction, with most independent variables yielding a substantive share of statistically significant positive and negative coefficients depending on model specification. We compare the strengths and weaknesses of three distinct approaches to estimating the relative importance of different model specification choices: nearest 1-neighbor; logistic; and deep learning. All three approaches reveal that the impact of the covariate space is relatively modest compared to the impact of sample selection and dependent variable operationalization. We conclude that model uncertainty stems more from sampling and measurement than conditioning and discuss the methodological implications for how to assess model uncertainty in the social sciences.