A novel computational approach for survival prediction via fusion of whole-slide images and clinical data.

Abstract

e15119 Background: Annotations free survival prediction based on whole-slide-images(WSIs) requires diverse semantic information extraction ranging from the tumor cellular scale, histological scale to microenvironment scale, which is however quite challenging for most existing single resolution based method. Furthermore, considering that clinical data usually serve as important references for oncologists in survival assessment, we endeavor to leverage the inherent multi-resolution characteristic of WSIs for cross-scale feature extraction and simultaneously explore its fusion with clinical data, in order to provide a novel survival prediction perspective derived from multimodal biomedical data for the oncologist. Methods: We explore a dual-stream cross-attention model based on the fusion of clinical data and WSIs for more accurate survival prediction. In the proposed model, highly discriminative patches for survival prediction are selected by constructing dynamic graphs for patches of different resolutions, then a dual-stream structure combined with a cross-attention mechanism is designed to extract cross-scale features. Besides, in the constructed clinical assistance module, clinical data and WSI patch features are fused to explore intrinsic cross-modal correlations and convey potentially complementary information. Results: WSIs and clinical data of 518 KIRC samples collected from The Cancer Genome Atlas (TCGA) are randomly divided into training, validation, and testing sets in a ratio of 7:1:2. The performance of the proposed model is evaluated using 5-fold cross-validation. The results are compared with other state-of-art methods. Abundant experiments have demonstrated that, compared with single-modal method, our multimodal method achieves a discernible performance improvement（1.65% performance improvement compared with TransMIL and 6.12% performance improvement compared with ILRA）with an overall c-index of 0.676, suggesting its potential superiority. Conclusions: We have developed a dual-stream cross-attention model based on the fusion of clinical data and WSIs for survival prediction, which can be an effective supplementary diagnostic tool for oncologists and pathologists. Furthermore, the abundant experimental results have demonstrated the positive impact of cross-scale semantic representation as well as the effective complementarity of clinical data for WSIs.