Cross-cancer homologous recombination deficiency prediction from whole slide images using transfer learning

Abstract

Abstract Accurate prediction of Homologous Recombination Deficiency (HRD) is vital for personalized cancer therapy, yet genomic assays are often costly and complex. Predicting HRD from Hematoxylin and Eosin (H&E) stained whole-slide images (WSIs) via deep learning offers an alternative, but model generalization across diverse cancer types remains challenging. We developed a three-stage transfer learning framework to improve HRD prediction across cancer types. After establishing baseline models on data from eight cancer types, the TCGA-BRCA model was selected as a pretrained model for transfer learning applied to eligible cohorts. Transfer learning enhanced HRD prediction performance specifically in cancers with histological similarity to the TCGA-BRCA source model. The AUC increased by + 4% (TCGA-LUAD), + 5% (CPTAC-LUAD), and + 1% (SARC) compared to baseline models. Interpretability analysis confirmed that the model’s predictions were driven by histologically relevant tumor regions, which was further supported by quantitative cellular analysis showing that HRD-high status was characterized by a significantly higher density of neoplastic cells (HRD-high: median 16.0, IQR 1.0–27.0 vs. HRD-low: median 1.0, IQR:0.0–5.0; p  < 0.001), whereas HRD-low cases featured a stromal-rich microenvironment. Conclusions This study demonstrates that transfer learning can enhance HRD prediction from H&E images for cancer types sharing histopathological features with the TCGA-BRCA source model, offering a more efficient and accessible approach for clinical HRD assessment.