News Hotspots Detection and Tracking Based on LDA Topic Model

With the rapid spread of Internet and the mobile web, the number of news pages is increasing quickly as well as the content of news becomes highly dynamic. It's difficult for normal users to obtain specific information contained in a mass of news streams. So it's of great research significance to study how to analyze massive news, detect and track news hotspots automatically. This research proposes to apply LDA (Latent Dirichlet Allocation) model to the application of topic detection and tracking. The news articles collected by crawlers are modeled by the LDA model in a form of document-topic-word distribution. We propose a method to compute the heat of topics based on the distribution and to detect the news hotspots. In addition, we track the evolution of the topic trends in different time-slices. Jenson-Shannon distance is used to measure the similarity between topics to identify topic inheritance and topic mutation. We conducted experiments on a dataset consisting of 3462 news texts from news portals. The result revealed that the proposed model has a good effect both in detecting hotspots and discovering meaningful topical evolution trends.