Grouped multi-scale vision transformer for medical image segmentation

Abstract

Abstract Medical image segmentation plays a pivotal role in clinical diagnosis and pathological research by delineating regions of interest within medical images. While early approaches based on Convolutional Neural Networks (CNNs) have achieved significant success, their limited receptive field constrains their ability to capture long-range dependencies. Recent advances in Vision Transformers (ViTs) have demonstrated remarkable improvements by leveraging self-attention mechanisms. However, existing ViT-based segmentation models often struggle to effectively capture multi-scale variations within a single attention layer, limiting their capacity to model complex anatomical structures. To address this limitation, we propose Grouped Multi-Scale Attention (GMSA), which enhances multi-scale feature representation by grouping channels and performing self-attention at different scales within a single layer. Additionally, we introduce Inter-Scale Attention (ISA) to facilitate cross-scale feature fusion, further improving segmentation performance. Extensive experiments on the Synapse, ACDC, and ISIC2018 datasets demonstrate the effectiveness of our model, achieving state-of-the-art results in medical image segmentation. Our code is available at: https://github.com/Chen2zheng/ScaleFormer.