MedZeroSeg: Zero-shot medical image segmentation via vision foundation models

Abstract

A novel medical image segmentation framework, MedZeroSeg , is proposed to address key challenges in the field. Leveraging vision foundation models such as CLIP (Contrastive Language-Image Pre-training) and SAM (Segment Anything Model), it achieves zero-shot segmentation, accurately delineating previously unseen medical images without requiring additional labeled data. This significantly reduces reliance on large-scale annotated datasets. At its core, MedZeroSeg introduces a Dual-Path Feature Extraction Module that captures both fine anatomical details and global contextual information through the integration of local and global perception mechanisms, enhancing robustness against the complexity and variability inherent in medical imaging.Additionally, a Context-Enhanced Hard-Negative Contrast Loss is introduced to enhance contrastive learning by exploiting contextual cues and refining hard-negative sampling, leading to better representations and higher efficiency. The key innovation of MedZeroSeg lies in its ability to leverage generalizable knowledge from CLIP and SAM without any task-specific fine-tuning, making it highly adaptable across different medical imaging modalities. Extensive experiments on three publicly available datasets, including cardiac MRI (ACDC), multi-organ abdominal CT (Synapse), and chest X-ray (COVID-QU-Ex), demonstrate that MedZeroSeg achieves superior results in both zero-shot and weakly supervised segmentation settings, showcasing strong generalization capabilities and minimal data dependency. The framework represents a significant advancement in medical image analysis and opens up promising directions for future research in applying advanced foundation models and innovative learning strategies to healthcare applications.