Domain-agnostic weakly supervised surgical instrument segmentation

Abstract

Abstract Recent advancements in visual foundation models open new avenues in the field of surgical instrument segmentation in medical images. Segmentation foundation models provide high segmentation accuracy for objects of interest that are selected via prompts in the form of points, bounding boxes, or text. However, the choice of suitable prompts either requires manual interaction or relies on two-stage pipelines based on supervised, typically domain-specific models. This limits their applicability for domain-agnostic surgical instrument segmentation. We propose a method for surgical instrument segmentation that leverages the power of the segmentation foundation model SAM2 while eliminating the need for a user-defined input prompt or domain-specific annotated datasets. We achieve this by utilizing an anomaly detector generated from non-instrument images to identify instruments as unseen regions and in this way, define a SAM2 input prompt based solely on image-level annotations. For three datasets for surgical instrument segmentation from diverse domains (EndoVis2017, CaDIS, and PASO-SIS), we achieve mean Normalized Surface Distances ranging from $${53}{\%}\,\hbox {to}\,{73}{\%}$$ . This demonstrates the competitiveness of our method compared to alternatives, while its training- and mask-free nature makes it well-suited for surgical workflow integration. By simplifying surgical instrument segmentation, we advance the field of computer-assisted surgery and unlock a wide variety of assistance functions with minimal effort.