Abstract 4370225: Optimizing nursing team assessment with computer vision and machine learning: A feasible approach for interstage video analysis

Abstract

Introduction: Despite decreases in mortality, infants with single-ventricle heart disease remain at significant risk for morbidity-associated “red flag” events during the interstage period. As patients are typically discharged home during this time, preventive care is essential through proactive nursing assessments of remote (parent entered) monitoring data. Preemptive identification of complex cyanotic complications remains challenging to identify from vital signs alone. Patient videos capturing movement, color, and status holds potential to enhance nursing assessments in asynchronous remote patient monitoring. Assessing subtle measures of risk from video data in a non-trivial task. Hypothesis: In addition to physiologic data, computer vision machine learning (ML) approaches applied to videos are a feasible method for aiding proactive, personalized video review of parent-obtained, interstage infant characteristics by a nursing care team. Methods: A retrospective multi-site cohort was obtained from the CHAMP® repository (3/2014 – 12/2022), including infants with at least one video prior to Glenn surgery or death. For each eligible video, thirty-three 3D pose landmarks of major body points were detected using MediaPipe, an open source pose mapping toolkit. Processed data was used to train a long short-term memory (LSTM) model pipeline to predict if an event occurred within 28 hours of the video upload time. Results: Infants (n=494)- demographics in Table 1- from 10 institutions with 4,858 candidate videos had a computer-vision and ML pipeline successfully applied. The team was able to extract, process, and score event risk from parent uploaded videos. Each video was ranked by the likelihood of experiencing an event and performance was evaluated using lift, focused on identifying events relative to random. However, the LSTM model, trained solely on pose landmarks, offered no improvement for identifying imminent red flag events. Conclusion(s): The ability to successfully capture pose and movement data from parent videos was confirmed and proves to be a promising adjunct to a full remote nursing assessment to augment parent-only reported red flags for high-risk congenital heart disease patients. The low predictive power of red-flag events alone encouraged current work to incorporate vitals signs, demographics, facial landmark features, respiratory effort, and skin tone to determine if the ML model can be further trained to aid in prioritizing video review.