CACE closed: A multiverse examination of the influence of implementation variability on student outcomes in a randomised controlled trial of a universal, school-based social-emotional learning intervention

Abstract

Introduction Amidst calls for more high-quality research to assess the influence of implementation variability on student outcomes in school-based trials using instrumental variable approaches (e.g., complier average causal effect; CACE), unreported researcher degrees of freedom can limit replicability and lead to uncertainty of conclusions. The current study aims to acknowledge and address these limitations using a multiverse framework. We investigate whether, and how, conclusions of a universal social-emotional learning intervention’s efficacy are contingent on decisions about how compliance is defined and modelled. Methods Secondary analysis of data from a cluster randomised control trial of the intervention Passport was undertaken, with schools (k = 62, N = 2,425 children) randomly allocated to intervention ( k  = 33; N = 1,264) or control ( k  = 29; N = 1,161) conditions. Ten theoretically plausible specifications for the CACE model were identified and pre-registered, including five definitions of compliance (fidelity, dosage, quality, responsiveness, reach) and two compliance thresholds (50 th and 75 th percentile). Student relational outcomes (bullying, peer support, loneliness) were assessed pre- and post-intervention. Results Multilevel intent-to-treat analysis revealed null intervention effects. Applying a multiverse framework to CACE revealed variation in model results, manifest in entropy values, the precision of confidence intervals and the direction, size and statistical significance of CACE effects. A statistically significant and negative CACE effect was found for peer support when compliance was defined by reach using the 75 th percentile (β = −.38, 95% CI (−.68, −.08), E = .71, d = −.21), but non-statistically significant intervention effects were observed for the remaining CACE models. Conclusions A multiverse framework enables transparent reporting of analytic uncertainty in evaluations of implementation variability in school-based trials, thereby offering theoretical, methodological and empirical advancements for implementation science. In doing so, it enables us to move from a fragmented view towards a more coherent understanding of complex interventions in real-world settings. Pre-registration www.osf.io/s5pmw .