Towards Application of Speech Analysis in Predicting Learners' Performance

In this work in progress, we propose a model for analysis of students' verbal conversation during teamwork to predict their academic performance based on expressed emotions. Our previous studies support the link between an individual's attitude and emotional states during the cognitive process with their performance in the given context [1], [2]. Traditionally the learners' affective states were assessed by having them fill out standard surveys. More recently the researchers have been using advanced methods to extract students' emotions from their writings by using Natural Language Processing (NLP) models. These models are applied to data collected from different sources such as discussion forums, team chats, students' reflective surveys, and journals. In this research, we take one step further by recording students audio in class as they converse about the course topic in low-stake teams and extract emotions from their conversations by NLP methods. The main contributions of the proposed model are 1) the audio transcription component 2) the multi-class emotion analysis unit and 3) the performance prediction model based on input data. SpeechBrain pre-trained models with transformer language models were applied for automated transcription of audio data and converting them to embedding vectors. NLP methods were applied for sentiment analysis. Next, we formed the feature set by combining the extracted emotions with students' formative assessment grades during the semester to implement a prediction model. We further analyzed which features in the feature set have a higher impact on the students' academic performance. The early result of this research is promising as we found high accuracy in the predicted scores of the students.