Gait recognition using spatio-temporal representation fusion learning network with IMU-based skeleton graph and body partition strategy

Abstract

The precise recognition of human lower limb movements based on wearable sensors is very important for human-computer interaction. However, the existing methods tend to ignore the dynamic spatial information in the process of executing human lower limb movements, leading to challenges such as reduced decoding accuracy and limited robustness. In this paper, we construct skeleton graph data based on inertial measurement unit (IMU) sensors. Also, a two-branch deep learning model, termed TCNN-MGCHN, is proposed to mine meaningful spatial and temporal feature representations from IMU-based skeleton graph data. Firstly, a temporal convolutional module (consisting of a multi-scale convolutional sub-module and an attention sub-module) is developed to extract temporal feature information with highly discriminative power. Secondly, a multi-scale graph convolutional module and a spatial graph edges’ importance weight assignment method based on body partitioning strategy are proposed to obtain intrinsic spatial feature information between different skeleton nodes. Finally, the fused spatio-temporal features are passed into the classification module to obtain the predicted gait movements and sub-phases. Extensive comparison and ablation studies are conducted on our self-constructed human lower limb movement dataset. The results demonstrate that TCNN-MGCHN delivers superior classification performance compared to the mainstream methods. This study can provide a benchmark for IMU-based human lower limb movement recognition and related deep-learning modeling works.