Bimodal sentiment analysis in social media: a one-shot learning approach

With the rapid growth of social media (forums, blogs, social networks, etc.) entities like polling organizations, advertising companies, as well as democratic organizations are seeking to discover the opinions of their audiences more than ever. Analyzing such information helps them to perform operations like production monitoring, market forecasting, and civil society demands with high accuracy. Besides these entities, individuals/customers also choose from other users' feedback. The most basic processing that takes place in the field of sentiment analysis is to classify a text into a positive or negative class. Yet, given to wide usage of image-based social media, there have been limited attempts to deduce emotions from images or videos. In this paper, a bimodal sentiment analysis is presented by combining visual and textual information. In the proposed method, the image is sentimentally captioned by an encoder-decoder network, first. To achieve this goal, a sentiment image captioning method has been used to generate positive, negative, or neutral captions for each input image. Then, the generated captions and the user's published text are sent to the relation network and then the final sentiment label is predicted. In this one-shot learning approach, only one image is used to predict a sentiment, without any training. Results show the proposed method can compete with the State-of-the-Art methods in the field of visual sentiment analysis. In this respect, the proposed method achieves 70% accuracy on TWITTER and 75% accuracy on MVSA datasets, which can be regarded as a champion considering that this method is a one-shot approach.