A rubric for assessing conformance to the Ten Rules for credible practice of modeling and simulation in healthcare

Abstract

The power of computational modeling and simulation (M&S) is realized when the results are credible, and the workflow generates evidence that supports credibility for the context of use. The Committee on Credible Practice of Modeling & Simulation in Healthcare was established to help address the need for processes and procedures to support the credible use of M&S in healthcare and biomedical research. Our community efforts have led to the Ten Rules (TR) for Credible Practice of M&S in life sciences and healthcare. This framework is an outcome of a multidisciplinary investigation from a wide range of stakeholders beginning in 2012. Here, we present a pragmatic rubric for assessing the conformance of an M&S activity to the TR. This rubric considers the ability of an M&S study to communicate how well the study conforms to the Ten Rules for credible practice and facilitate outreach to a wide range of stakeholders from context-specific M&S practitioners to policymakers. It uses an ordinal scale ranging from Insufficient (zero) to Comprehensive (four) that is applicable to each rule, providing a uniform approach for comparing assessments across different reviewers and different modeling studies. We used the rubric to evaluate the conformance of two computational modeling activities: 1. six viral disease (COVID-19) propagation models, and 2. a model of hepatic glycogenolysis with neural innervation and calcium signaling. These examples were used to evaluate the applicability of the rubric and illustrate rubric usage in real-world M&S scenarios including those that bridge scientific M&S with policymaking. The COVID-19 M&S studies were of particular interest because they needed to be quickly operationalized by government and private decision-makers early in the COVID-19 pandemic and were accessible as open-source tools. Our findings demonstrate that the TR rubric represents a systematic tool for assessing the conformance of an M&S activity to codified good practices and enhances the value of the TR for supporting real-world decision-making.