A multimodal emotion classification method considering micro-expression information and simulating human visual attention mechanism

Micro-expressions, as fleeting and involuntary facial expressions, genuinely reflect an individual's emotional state. However, the characteristics of micro-expressions, primarily manifested in subtle movements of specific facial regions, affect the accuracy of recognition. Moreover, micro-expression recognition methods relying solely on visual information are limited. To address these issues, this paper proposes a Periphery Attention Fusion Network (PAFN) for micro-expression recognition. The method integrates facial expressions, electroencephalogram (EEG) time series, and spatial sequence information, aiming to enhance the accuracy and reliability of micro-expression recognition through multimodal information fusion. PAFN consists of three key modules: the Three-Dimensional Construction Module (3DCM) for constructing the three-dimensional features of EEG signals; the Preliminary Convolutional Preprocessing Module (PCP) applying dynamic convolution and depthwise separable convolution techniques for preliminary extraction of facial features; and the Periphery Self-Attention Memory Module (PSAM) combined with Long Short-Term Memory (LSTM) networks, generating peripheral positional encoding and adjusting attention weights to simulate the human visual attention mechanism, reducing interference and focusing on key information. Experimental results on the DEAP dataset and the microexpression PEG dataset demonstrate that the PAFN method's recognition accuracy exceeds 96%, outperforming existing methods, confirming its efficiency and advancement in the field of micro-expression recognition.