Automatic report generation based on multi-modal information

In this paper, we propose a new framework which can utilize multi-modal social media information to automatically generate related reports for users or government. First, we utilize DBSCAN (Density Based Spatial Clustering of Applications with Noise) to detect events in official news websites. Then, some unofficial information details are extracted from social network platforms (Foursquare, Twitter, YouTube), which will be leveraged to enhance the official report in order to excavate some latent and useful information. In this process, we applied some classic textual processing methods and computer vision technologies to reduce the noise information uploaded by user generated contents (UGCs). Then, we applied LSTM-CNN model to generate the related image caption and successfully convert visual information to textual information. Finally, we extracted some latent topics using graph cluster method to generate the final report. To demonstrate the effectiveness of our framework, we got a large of multi-source event dataset from official news websites and Twitter. Finally, the user study demonstrates the practicability of our approach.