Hybrid Big Data Architecture for High-Speed Log Anomaly Detection

Log processing can be very challenging, especially for environments with lots of servers. In these environments, log data is large, coming at high-speed, and have various formats, the classic case of big data problem. This makes anomaly detection very difficult due to the fact that to get good accuracy, large amount of data must be processed in real-time. To solve this problem, this paper proposes a hybrid architecture for log anomaly detection using Apache Spark for data processing and Apache Flume for data collecting. To demonstrate the capabilities of our proposed solution, we implement a SARIMA-based anomaly detection as a case study. The experimental results clearly indicated that our proposed architecture can support log processing in large-scale environment effectively.