Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition

In this paper, a pen-based 3D hand gesture dataset and recognition method using ultrasonic positioning and inertial data is proposed. First, considering that 3D hand gestures have six degrees of freedom, a 3D hand gesture dataset based on trajectory shape attributes, motion direction attributes and pen attitude attributes is proposed. Then, each attribute of the gesture is processed according to its priority, and the corresponding data channel and recognition method are selected to determine the 3D hand gesture label. Finally, experimental verification is conducted using a 3D multi-channel pen-like interactive device. For a 10-gesture set, the gesture recognition rates achieved ranged from 86.5–99.5%, depending on whether a single or multiple templates and thresholds are used. The results show that the 3D hand gesture recognition method proposed in this paper can recognize pen-based gestures effectively and solve the problem of traditional gesture recognition methods not being able to recognize 3D hand gestures containing multiple attributes.