Optimising the Delphi survey method during core set development: The impact of summarised feedback on stakeholders’ prioritisation of core data items and consensus

Abstract

Background Core outcome sets are an established method for standardising the collection, measurement and reporting of treatment outcomes in effectiveness trials. Using Delphi survey methodology, core sets are developed by prioritising and re-prioritising data items facilitated by provision of feedback of other stakeholders’ responses. It is unknown how best to provide feedback to ensure that it influences the re-prioritisation of items effectively. This study examined whether informing participants of the top-rated items from the previous survey round may influence the re-prioritisation of data items in a subsequent survey round during the development of core data sets. Methods This study was nested in the development of a registry core data set. In round two of the Delphi survey, participants were randomised to receive ‘standard’ or ‘enhanced’ instructions. ‘Enhanced’ instructions included summarised data of the top five data items scored by participants in the previous survey round) in addition to standard feedback (the median round 1 score per item). Items scored 7–9 by ≥70% of participants in round 2 were considered ‘prioritised’. Concordant/discordant items were determined and extent of agreement between groups calculated (kappa statistics). Results Both groups prioritised a larger number of items in round 2 than in round 1 and there was little difference in the percentage of respondents prioritising the ‘Top 5’ items in round 2 (mean change in prioritisation of Top 5 items for all four core sets combined – 2.3% increase in standard group and 3.2% increase in enhanced group). Overall agreement in data items prioritised by both groups improved in round 2 (discordant items – 11% in round 1 and 4% in round 2). Conclusion Providing participants with additional feedback during the process of item prioritisation did not promote prioritisation of items during development of a core set. In the development of health core sets, where often many items are prioritised, further work to determine how to clearly and optimally communicatee feedback in a manner that promotes consensus effectively is required. Specifically, qualitative work with relevant stakeholders, exploring and clarifying the concepts of prioritisation and consensus, is warranted.