Predicting Users' Mental Effort in Drawing Tasks Using Gesture Recognition

A new frontier in human-centered AI is the development of systems that can understand the mental effort of their users in order to better support them in their day-to-day lives. Current solutions rely on specialized sensors to gain insight into a person's cognitive load by measuring psychophysiological signals, limiting opportunities for wide-scale adoption. However, behavioral signals such as user interactions provide a more convenient medium to understand a person's thought processes, so it is possible to use gesture recognition to predict a person's mental effort. To assess the feasibility of gesture-based cognitive load estimation, this work will investigate features in pen gestures that are predictive of a person's mental effort. The goals of this work include the design and development of an intelligent pen-based interface that can predict the user's mental effort in an explainable fashion. To ensure the validity of the results, this work focuses on drawing tasks related to spatial visualization to ground the findings in the cognitive skills of the user.