Validation of LungFlag Lean machine-learning model to identify individuals with lung cancer using multinational data.

Abstract

e13649 Background: Lung cancer (LC) is the leading cause of cancer-related deaths worldwide. Risk prediction models developed to identify elevated risk populations. LungFlag (LF) is ML model based on routine historical EHR data, validated and in clinical practice since 2023. LungFlag Lean (LFL) is ML-developed risk prediction uses minimal patient-level data (demographics, vitals, smoking, diagnoses, labs) from questionnaire or EHR to identify individuals at higher risk for LC, prompting HCP to recommend further assessment. One of prominent hurdles for EHR based risk models are data incompleteness and/or quality issues, which can be addressed by LFL, designed all inputs could easily filled by HCP with simple questions for an individual and/or by review of medical records. Methods: The robustness of the LFL was validated by applying it to data from multiple large-scale data sources. Performance of LFL on ever smokers 40-89 compared to 2 models: mPLCO 2012 (PLCO) based on questionnaire, and LF uses EHR data. Comparison was carried out retrospectively on three independent US-based datasets and one dataset from the UK by using a bootstrapping analysis. Performance was evaluated by AUC, sensitivity/specificity and odds ratio for two potential cut points for 'high risk' individuals, one selected for high specificity (3% positivity rate) and one for high sensitivity (70% sens). Difference between models was measured by calculation of p-value. Results: Analysis included data from 2.2M eligible individuals (99K LC cases), extracted from 4 data sets. Performance demonstrated that the overall AUC of LFL ranges between .800 and .839 (compared to .812-.843 for LF and .797-.836 for PLCO). The sensitivity and odds ratio at the top 3% were better than the PLCO and slightly lower than LF. The same trend was demonstrated for the cut-off associated with the 70% sensitivity level. Noted that the average age of the population flagged by the model was > 2 years younger than the population flagged by the PLCO and slightly older than that of LF. Conclusions: LFL demonstrated a consistent performance across multiple data sources. Additionally, it demonstrated performance comparable and slightly better than the existing PLCO model and flagged younger individuals. Therefore, it can be concluded that LFL may provide an alternative risk prediction model when limited data is available. Source Model Cases Controls AUC 3% Positivity Rate 70% Sensitivity OR Sens% OR Spec% KPSC LF 4,993 75,084 .814 9.3†† 21.9†† 7.8 77.1 LFL .817 8.0 19.5 8.1 77.6 PLCO .803† 7.7 18.8 7.3† 75.7† GHS LF 1,889 170,399 .812 9.5†† 22.5†† 7.9†† 77.1†† LFL .800 7.3 18.3 6.8 74.4 PLCO .798 6.8 17.3 6.7 74.0 US3 LF 80,696 338,678 .812 †† 10.4 †† 23.9†† 7.5†† 76.4†† LFL .800 8.5 20.6 6.6 74.4 PLCO .797 † 6.9† 17.4† 6.7 74.1† THIN LF 11,766 1,521,573 .843 †† 10.4 †† 24.0 †† 10.3 †† 81.5†† LFL .839 9.7 22.8 9.8 80.4 PLCO 0.836 † 9.0† 21.5† 9.4† 80.1 †PLCO and LFL (p <.05). ††LF and LFL (p <.05).