An Approach for Scale Suspicious Network Events Detection

Detecting the real suspicious events from a large number of low-quality alerts is a severe challenge to the security operations center teams. In this paper, we present an approach to this problem by following the sequence of machine learning steps. The highlight of our approach is the method to generate two simple but effective categories of features based on group and aggregation operations, which can scale with a large number of alerts using MapReduce framework. The two generated types of features are local features and global features. The local features cover the alert aggregation information of the same group of events, while the global features cover the network aggregation information of different groups of events. Moreover, we also introduce the model stacking mechanism to enhance the robustness of the model. The proposed approach achieves AUC scores of 0.9512 on the validating dataset and 0.9303 on the test set, which is the 2nd highest final score in the competition.