A weakly supervised method for surgical scene components detection with visual foundation model

Abstract

Purpose: Detection of crucial components is a fundamental problem in surgical scene understanding. Limited by the huge cost of spatial annotation, current studies mainly focus on the recognition of three surgical elements ⟨instrument, verb, target⟩, while the detection of surgical components ⟨instrument, target⟩ remains highly challenging. Some efforts have been made to detect surgical components, yet their limitations include: (1) Detection performance highly depends on the amount of manual spatial annotations; (2) No previous study has investigated the detection of targets. Methods: We introduce a weakly supervised method for detecting key components by novelly combining the surgical triplet recognition model and the foundation model of Segment Anything Model (SAM). First, by setting appropriate prompts, we used SAM to generate candidate regions for surgical components. Then, we preliminarily localize components by extracting positive activation areas in class activation maps from the recognition model. However, using instrument’s class activation as a position attention guide for target recognition leads to positional deviations in the target’s resulting positive activation. To tackle this issue, we propose RDV-AGC by introducing an Attention Guide Correction (AGC) module. This module adjusts the attention guidance for target according to the instrument’s forward direction. Finally, we match the initial localization of instruments and targets with the candidate areas generated by SAM, achieving precise detection of components in the surgical scene. Results: Through ablation studies and comparisons to similar works, our method has achieved remarkable performance without requiring any spatial annotations. Conclusion: This study introduced a novel weakly supervised method for detecting surgical components by integrating the surgical triplet recognition model with visual foundation model.