Application of multiscale token fusion and pruning in CNN–Transformer hybrids under low-data training for image recognition

Abstract

Abstract In this paper, a hybrid convolutional neural network (CNN) and transformer architecture for image classification that explicitly exploits multiscale spatial representations while maintaining computational efficiency is proposed. A convolutional backbone is first used to extract hierarchical feature maps, which are subsequently tokenized and fused through a cross-scale token fusion (CSTF) mechanism. In addition, several token-level and CNN-level pruning strategies are evaluated to examine whether redundant spatial tokens or convolutional features can be removed without substantially degrading performance. Extensive experiments on Caltech-101 and Oxford-IIIT Pets under low-data training conditions show that the best proposed configurations are dataset-dependent: the single 14 $$\:\times\:14$$ scale achieves 70.28 $$\:\pm\:1.91$$ % accuracy on Caltech-101, while the single 7 $$\:\times\:$$ 7 scale achieves 28.35 $$\:\pm\:1.62$$ % accuracy on Oxford-IIIT Pets. Among the multi-scale fusion models, 7 $$\:\times\:$$ 7+14 $$\:\times\:$$ 14 performs best on Caltech-101 69.68 $$\:\pm\:$$ 1.77%, whereas 7 $$\:\times\:$$ 7+14 $$\:\times\:$$ 14+28 $$\:\times\:$$ 28 performs best on Oxford-IIIT Pets 27.57 $$\:\pm\:$$ 1.05%. CNN kernel pruning achieves the strongest pruning performance on both datasets, reaching 68.37 $$\:\pm\:$$ 1.06% on Caltech-101 and 28.15 $$\:\pm\:$$ 1.15% on Oxford-IIIT Pets. These results indicate that multi-scale token fusion can provide competitive performance, but careful scale selection is more important than simply increasing the number of spatial token scales.