The apple detection method based on multimodal features

Abstract

Accurate detection of apples and other fruits in complex environments remains a formidable challenge due to the intricate interplay of varying lighting conditions, occlusions, and background clutter. Traditional detection methods, which primarily rely on RGB images or incremental improvement of deep learning models, often fail to achieve satisfactory detection accuracy. To address this, an innovative method of apple detection is proposed to improve the detection performance through multimodal feature fusion rather than radical architectural modifications. The proposed method integrates four complementary modalities: RGB image, color and edge feature maps, depth feature map, and point clouds. Chromatic properties of fruits and geometric boundaries of fruit-tree structures are captured by color and edge feature maps extracted from RGB inputs, which are weighted and fused into a composite feature channel. The depth map and point clouds acquired via binocular active infrared stereo cameras provide additional spatial information. The depth feature image is used as a standalone feature channel. Given the significant modal discrepancies between point clouds and RGB data, a preprocessing pipeline is implemented: voxel sampling and local anomaly detection are first applied to denoise and fill holes in point clouds, followed by recalibrated mapping to ensure spatial alignment with RGB image. The XYZ coordinates of processed point clouds are then used as three distinct feature channels. Finally, the YOLOv5 input layer is redesigned to accept multi-channel feature inputs.Multimodal fusion enriches the feature representation accessible to YOLOv5, enhancing model robustness against lighting variations and background noise. Experimental results demonstrate that the proposed method achieves 95.8% precision (P), 96.0% recall (R), and 95.9% F1-score in complex scenarios. Compared to baseline methods using RGB-only and RGB+depth inputs, precision improvements of 7.4% and 6.3% are observed respectively.