Never Correct: The Novel Analysis of Differing Visual (Facial Expression) and Acoustic (Vocalization) Bimodal Displays of the Affective States "Pain", "Pleasure", and "Neutral"

In real-world scenarios, humans estimate a large proportion of their perceived world contextually and use previous information to adjust or modify their expectations and responses. A typical example of congruence is when a smile (visual modality) is accompanied by a laugh (acoustic modality). Pain and pleasure are extremely intensive affects, rarely correctly assessed. It rarely happens that affective communication is incongruent in different modalities. Intentional combinations of these two affective expressions may be implemented by using audiovisual media. It is to be expected that presentations involving incongruent combinations during gaming and other interactions with machines will alter perceptions and the impact of ambiguity. To evaluate the impact of sensory crosstalk a novel statistical analysis was developed to estimate the impact of each modality. The results show that the visual modality dominates for the pairing pain/pleasure for. The acoustic modality dominates for pain/pleasure, pain/neutral, and pleasure/neutral. For neutral/pain and neutral/pleasure, neither the visual nor the acoustic modality enables highly correct ratings. The findings are of high value to the psychology of perception on a theoretical level and game/media developers as application fields.