PoulTrans: a transformer-based model for accurate poultry condition assessment

Abstract

Abstract Recent advances in deep learning have significantly enhanced the accuracy of poultry image recognition, particularly in assessing poultry conditions. However, developing intuitive decision support tools remain a significant challenge. To address this, we present PoulTrans, an innovative image captioning framework that leverages a Convolutional Neural Network (CNN) integrated with a CSA_Encoder-Transformer architecture to generate detailed poultry status reports. This model incorporates visual features extracted by CNNs into the Channel Spatial Attention Segmentation Encoder (CSA_Encoder), which produces segmented channel and spatial attention outputs. To optimize multi-level attention and improve the semantic precision of the status descriptions, we introduced a Channel Spatial Memory-Guided Transformer (CSMT) and a novel PS-Loss function. The performance of PoulTrans was tested on the PSC-Captions dataset, achieving top scores of 0.501, 0.803, 4.927, 0.608, and 1.882 for the BLEU-4, ROUGE-L, CIDEr, SPICE, and Sm metrics, respectively. Comprehensive analyses and experiments have validated the effectiveness and reliability of our model, providing advanced tools for automated poultry status generation and enhancing the digital experience for poultry farmers. Our code is available at: https://github.com/kong1107800/PoulTrans.