HAVIT: research on vision-language gesture interaction mechanism for smart furniture

Abstract

Abstract With the rapid development of smart furniture, gesture recognition has gained increasing attention as a natural and intuitive interaction method. However, in practical applications, issues such as limited data resources and insufficient semantic understanding have significantly constrained the effectiveness of gesture recognition technology. To address these challenges, this study proposes HAVIT, a hybrid deep learning model based on Vision Transformer and ALBEF, aimed at enhancing the performance of gesture recognition systems under data-scarce conditions. The model achieves efficient feature extraction and accurate recognition of gesture characteristics through the organic integration of Vision Transformer’s feature extraction capabilities and ALBEF’s semantic understanding mechanism. Experimental results demonstrate that on a fully labeled dataset, the HAVIT model achieved a classification accuracy of 91.83% and an AUC value of 0.92; under 20% label deficiency conditions, the model maintained an accuracy of 86.89% and an AUC value of 0.88, exhibiting strong robustness. The research findings provide new solutions for the development of smart furniture interaction technology and hold significant implications for advancing practical applications in this field.