Robust Motion Recognition Using Gesture Phase Annotation

Robust gesture recognition is key to multimodal language understanding as well as human-computer interaction. While vision-based approaches to gesture recognition rightly focus on detecting hand poses in a single frame of video, there is less focus on recognizing the distinct "phases" of gesture as used in real interaction between humans or between humans and computers. Following the semantics of gesture originally outlined by Kendon, and elaborated by many such as McNeill and Lascarides and Stone, we propose a method to automatically detect the preparatory, "stroke," and recovery phases of semantic gestures. This method can be used to mitigate errors in automatic motion recognition, such as when the hand pose of a gesture is formed before semantic content is intended to be communicated and in semi-automatically creating or augmenting large gesture-speech alignment corpora.