A contactless method to measure real-time finger motion using depth-based pose estimation

Background: Finger mobility plays a crucial role in everyday living and is a leading indicator during hand rehabilitation and assistance tasks. Depth-based hand pose estimation is a potentially low-cost solution for the clinical and home-based measurement of symptoms of limited human finger motion. Objective: The purpose of this study was to achieve the contactless measurement of finger motion based on depth-based hand pose estimation using Azure Kinect depth cameras and transfer learning, and to evaluate the accuracy in comparison with a three-dimensional motion analysis (3DMA) system. Methods: Thirty participants performed a series of tasks during which their hand motions were measured concurrently using the Azure Kinect and 3DMA systems. We propose a simple and effective approach to achieving real-time hand pose estimations from single depth images using ensemble convolutional neural networks trained by a transfer learning strategy. Algorithms to calculate the finger joint motion angles are presented by tracking the locations of the 24 hand joints. To demonstrate their potential, the Azure-Kinect-based 3D finger motion measurement system and algorithms are experimentally verified through comparison with a camera-based 3DMA system, which is the gold standard. Results: Our results revealed that the Azure-Kinect-based hand pose estimation system produced highly correlated measurements of hand joint coordinates. Our method achieved excellent performance in terms of the fraction of good frames ( > 80%) when the error thresholds were larger than approximately 2 cm, and the range of mean error distance was 0.23 - - 1.05 cm. For joint angles, the Azure Kinect and 3DMA systems had comparable inter-trial reliability (ICC2,1 ranging from 0.89 to 0.97) and excellent concurrent validity, with Pearsons r-values > 0.90 for most measurements (range: 0.88 - - 0.97). The 95% BlandAltman limits of agreement were narrow enough for the Azure Kinect to be considered a valid tool for the measurement of all reported joint angles of the index finger and thumb in pinching. Moreover, our method runs in real time at over 45 fps. Conclusion: The results of this study suggest that the proposed method has the capacity to measure the performance of fine motor skills.