Automatic Detection of Irony Based on Acoustic Features and Facial Expressions

The current study deals with the automatic analysis of verbal irony using artificial neural networks. Detection of verbal irony is an important task nowadays, because the effectiveness of the communication depends on the correct interpretation of sentences with an ambiguous meaning. In the case, when the context is lacking, the correct sense can be understood not from the lexical content, but through phonetic features, as well as through co-speech mimics and gestures. Thus we accomplished a new research on the material of the multimedia corpus of Russian ironic speech, which contains the detailed phonetic annotation and irony evaluation by native listeners in perceptual auditory experiments. Two types of automated analysis were accomplished: based on acoustic feature and facial expression extraction. The use of the fully connected neural network and of the Wav2Vec 2.0 model for the automatic irony detection in audio signal demonstrated high level of irony recognition. We also tested on a part of the corpus the recognition of ironic facial expressions in video signal using convolutional neural network and the PyFeat library, which allowed us to conclude that this model can give good results when we increase the amount of the material.