Broad implementation of AI for lymph node assessment: Insights from a head-to-head comparison of two applications.

Abstract

e13679 Background: Lymph node (LN) assessment is pivotal for guiding treatment in breast cancer (BC), head- and neck cancer (HNC) and melanoma, yet it imposes a significant workload on pathologists and sometimes involves high costs (immunohistochemical stains), making it well-suited for AI-assistance. Here, we evaluate the performance of two CE-IVD certified AI-applications (DeepPath-LYDIA© (DP) and the Metastasis-Detection-App by Visiopharm© (VP)), both inside (IIU) and outside their intended use (OIU). Methods: Both apps were tested in positive LNs of ~100 patients for HNC (both OIU) and melanoma (DP IIU, VP OIU), and for BC (both IIU) in the 59 positive sentinel LN-samples (SLN) from the CONFIDENT-B trial. DP and VP highlight suspicious areas through colored outlines (“alerts”). Sensitivity and false alerts (FAs) were assessed. For BC this was assessed in 20 random negative SLN-cases (10 with-, and 10 without prior treatment) and for HNC and melanoma in up to 3 negative slides per patient. Results: Both apps detected all macro-metastases across tumor types (Table 1). For BC, both DP and VP detected all but one case of micro-metastases, which was undetectable on HE due to heavy cauterization. In contrast, isolated tumor cells (ITC), only relevant in case of neoadjuvant therapy, were detected in 8 of 18 cases. For HNC, DP performed excellent with 100% sensitivity for all metastases, whereas VP missed one case of micro-metastases and 2 of 3 ITC cases. For melanoma, DP missed one case of micro-metastases, while VP missed three cases. ITC-detection was only moderate for both (DP: 50.0%, VP: 62.5%). FAs for both apps were comparable in HNC and melanoma (average 8-9 per slide), whereas in BC, VP showed considerably more FAs (no prior therapy: average 8.4 vs. 4.0 for DP, neoadjuvant therapy: 17.4 vs. 6.8 for DP), which can mainly be explained by the method of annotation (more detailed versus broad outlines) and subsequent counting. Conclusions: Two commercially available AI-applications from different companies performed similar in the detection of LN micro- and macro-metastases in multiple tumor types, both IIU and OIU. For ITC, with clinical relevance depending on tumor type, performance was moderate in general. This may enable implementation of a single AI-solution for a broad indication, thereby positively impacting the business case for individual pathology laboratories. Sensitivity. Breast cancer (n=59) Macro-metastases (n=17) Micro-metastases (n=24) ITC (n=18)  DP (IIU) 100% (n=17) 95.8% (n=23) 44.4% (n=8)  VP (IIU) 100% (n=17) 95.8% (n=23) 44.4% (n=8) Head and neck cancer (n=100) Macro-metastases (n=75) Micro-metastases (n=22) ITC (n=3)  DP (OIU) 100% (n=75) 100% (n=22) 100% (n=3)  VP (OIU) 100% (n=75) 95.5% (n=21) 33.3% (n=1) Melanoma (n=98) Macro-metastases (n=66) Micro-metastases (n=24) ITC (n=8)  DP (OIU) 100% (n=66) 95.8% (n=23) 50% (n=4)  VP (OIU) 100% (n=66) 87.5% (n=21) 62.5% (n=5)