Anomaly Detection for Big Log Data Using a Hadoop Ecosystem

In this paper, we address a novel method to efficiently manage and analyze a large amount of log data. First, we present a new Apache Hive-based data storage and analysis architecture to process a large volume of Hadoop log data, which rapidly occur in multiple nodes. Second, we design and implement three simple but efficient anomaly detection methods. These methods use moving average and 3-sigma techniques to detect anomalies in log data. Finally, we show that all the three methods detect abnormal intervals properly, and the weighted anomaly detection methods are more precise than the basic one. These results indicate that our research is an excellent and simple approach in detecting anomalies of log data on a Hadoop ecosystem.