Enhancing Emotional Induction in Virtual Reality: Innovative Fusion of Speech and Realistic Non-Verbal Emotional Expression

Virtual reality (VR) provides an unparalleled platform for exploring the intricate interplay between non-verbal and verbal cues in emotion induction. However, the efficient implementation of non-verbal expressions in VR has often been hindered by a lack of rapid and realistic execution. In this paper, we address this gap by introducing an innovative processing pipeline that seamlessly integrates non-verbal cues to enhance the emotional impact of VR experiences. A novel processing pipeline was employed for non-verbal expression, utilizing a text-dictionary matching approach for body movements and audio-driven lip synchronization for realistic lip movements. Additionally, facial expressions were analyzed at the sentence level using GPT-generated prompts for sentiment analysis. Our experiment comprised two distinct groups: the Speech Group and the Speech+Motion Group, where participants not only heard the negative emotional speech but also witnessed corresponding virtual characters displaying synchronized body movements and facial expressions. Both groups exhibited adverse effects on positive emotions. Negative emotion induction in the Speech+Motion Group surpassed that of the Speech Group, particularly in the dimension of anger, reaching a statistically significant difference. Our findings underscore the impact of multimodal stimuli on emotional experiences in VR and introduce a processing pipeline that enables rapid and realistic integration of non-verbal cues.