Towards Explainable Visual Emotion Understanding

Emotion understanding from an image is an important computer vision research topic, but the reasoning behind the justification has been largely unexplored. We propose to utilize language, and more specifically, explanation of the judgement to increase the explainability for this task. We collect a dataset with image, emotion tag and explanation in natural language, and conduct analysis on the dataset to gain insights on the ambiguity of the human emotion perceptual. We examine baseline methods to predict emotion from image, explanation and general image description, and unifying both modalities. Our experiments shed lights on effects from different modalities and we also identify opportunities for future visual emotion categorization research based on the analysis. We release our dataset to advance future research.