Recognition of Infants' Gaze Behaviors and Emotions

This paper proposes a system for recognition of infants' gaze behaviors and emotions from videos. In the current work, researchers believed that the information of eye region is crucial for gaze behavior recognition, and emotion recognition mostly depends on the appearance of face. However, because the differentiation of all parts of infant's body has not finished yet, we found that infants always express their intentions and emotions using their whole body, especially moving their heads. Therefore, we incorporate the head pose information as features into the gaze behavior recognition, and we extract the gaze features to improve the recognition of infants' emotions. In addition, we combine several Deep Neural Networks, which can not only capture the details of the images very well, but also make full use of the temporal features. In order to recognize infants' gaze behaviors, we design a feature-extraction convolutional neural network which can obtain the features of infants' gaze direction, then we feed these features with head pose into the next gaze behavior recurrent neural network. Moreover, we combine the features of facial express and gaze behavior to characterize infants' emotions, and expand this system with an emotion recurrent neural network. In the end, we achieve the recognition accuracy 98.31% and 94.71% respectively on our data set.