Investigating the brain network characteristics of multimodal emotion recognition and its classification applications based on functional connectivity patterns

Emotion information can be expressed in multiple modal stimuli, and the brain can recognize multi-modal emotions efficiently and accurately. Recent researches have thoroughly analyzed the activity characteristics of relevant brain regions under different modal emotion information based on functional magnetic resonance imaging (fMRI). However, considering the functional integration characteristics of the brain in cognitive activities, further research on the brain network in the process of multi-modal emotion recognition should be carried out, which can reveal the brain's multi-modal emotion cognition mechanism more comprehensively. In this study, functional connectivity (FC) analysis was performed on the fMRI data from multimodal emotion recognition tasks. The correlation coefficients of brain regions were calculated and statistically analyzed to study the characteristics of FC patterns in multimodal emotion recognition processing. Moreover, the emotional information was decoded with different machine learning classification algorithms based on the FC patterns. The results showed that the modal and valence of emotion can lead to structural similarities and connection strength differences in connection strength of brain region connections in the brain network, and this property can successfully support the decoding of emotional information, and the decoding accuracy is higher than the previous decoding accuracy based on brain region activation patterns. This study explores the cognitive mechanism of multi-modal emotion recognition from a new perspective of brain functional integration, and compensates for the lack of brain signal decoding methods based on FC features and machine learning methods.