Visualizing vastness: Graphical methods for multiverse analysis

Abstract

Multiverse analysis is an increasingly popular tool for improving the robustness and transparency of empirical research. Yet, visualization techniques for multiverse analysis are underdeveloped. We identify critical weaknesses in existing multiverse visualizations—specification curves and density plots—and introduce a novel alternative: multiverse plots. Using both simulated and real-world data, we illustrate how multiverse plots can retain detailed information even in the face of thousands of model specifications. Multiverse plots overcome key limitations of existing methods by eliminating arbitrary sampling (a common issue with specification curves) and information loss on analytical decisions (an issue with density plots). Furthermore, they effectively show what conclusions a dataset can reasonably support and which researcher decisions drive variation in results. By providing software code to generate multiverse plots in Stata and R, we enable analysts to visualize multiverse results transparently and comprehensively.