A High Throughput Distributed Log Stream Processing System for Network Security Analysis

Computer-system logs often contain high volumes of interesting, useful information, and are an important data source for network security analysis. In this paper, we propose a distributed log stream processing system consisting of three main parts: log collection module, log transmission module and log statistics module. The system uses several open source technologies, not only supports multi-source heterogeneous log collection, but also provides near-real-time online statistics for log stream and offline statistics for massive log. In addition, we adopt a layered architecture in the log collection module, and accomplish a reliable Kafka consumer to get higher scalability as well as reliability. Using log entries generated by the network security platform as data source to do experiment, demonstrates that the proposed system is an effective and practical log stream processing system.