Joint model for learning state recognition with combining action detection and object detection

How younger students, such as primary and secondary school students, can improve their concentration when studying alone has been the subject of research by education experts. In this paper, we develop a learning-state-based tomato-clock system that can help improve concentration. We propose a joint model to detect students' learning states and thus control whether the tasks of the tomato-clock can be completed properly, which in turn motivates students to focus on their learning. In the joint model, the temporal action detection model SlowFast detects the video and identifies the base action category and the state action category. In cases where the actions are similar, such as the states of reading and watching video on mobile phone, we calculate the student's head pose information to determine his or her field of view and use the FPN-Faster RCNN model to detect the key items within his or her field of view to detect the real action. Finally, their learning state was identified based on the duration of the action, with mAP of 85.28%.