Comparison of deep learning-based three-dimensional human pose estimation methods with motion capture for gesture research

Abstract

Humans often make gestures while speaking. These gestures have been extensively researched in psychology and cognitive science, revealing their functions and roles in communication. Despite being produced in three-dimensional (3D) space, gestures and gesture space have primarily been measured and analyzed using two-dimensional planes. Optical motion capture (MoCap) is commonly used for 3D gesture measurement. However, MoCap is expensive, requires a large space, and the attachment of retroreflective markers can interfere with the natural generation of gestures by the speaker. Recent advances in deep learning-based human pose estimation (HPE) offer promising alternatives by enabling 3D keypoint estimation from standard video cameras. This study investigates the accuracy of four HPEs—two monocular and two stereo with triangulation methods—in estimating upper-body keypoints commonly used in natural scenarios for gesture research. Ten participants were recorded performing gesture-rich speech by both a MoCap system and three video cameras. We compared 13 keypoints, including the wrists, elbows, shoulders, fingers, and face, with MoCap data, using Euclidean distance as an error metric. Statistical analyses revealed that stereo methods significantly outperformed monocular methods across all keypoints. The most accurate method achieved an average error of 49.4 mm with respect to MoCap, suggesting sufficient accuracy for practical gesture analysis. Additionally, we visualized 3D gesture spaces and found a 75.4% overlap between HPE and MoCap at a voxel size of 50 mm, indicating high spatial agreement. Our findings demonstrate that stereo HPE methods are viable, cost-effective alternatives to MoCap in gesture-related applications. Our ultimate objective is to create a toolbox that is accurate and easy to use for measuring human gestures in 3D space by exploiting the recent advances in HPE as an alternative to MoCap. Such a toolbox will be suitable for non-experts in machine learning, and these results lay the foundation for it.