MSCA: An Unsupervised Anomaly Detection System for Network Security in Backbone Network

Anomaly detection is a crucial topic in network security which refers to automatically mining known and unknown attacks or threats. Many detectors have been proposed in the last decade. Nonetheless, a practical solution, which is able to provide a high True Positive Rate (TPR) with an acceptable False Positive Rate (FPR) without any prior information, is still challenging due to the complexity and variability of anomaly pattern. In this article, we propose a novel unsupervised detection system called MSCA which applies multiple sketches, K-means++ unsupervised clustering, and association rule mining to detect traffic anomalies and analyze anomalous features and correlations. It can blindly identify known and unknown traffic anomalies without any labeled traffic or prior signatures about data distribution. Rich traffic data is first aggregated and compacted to traffic flows by sketches, and further detected by the combination of clustering algorithm and voting strategy. Then association rule mining is finally utilized to find the anomalous frequent item-sets and association rules. Numerical experiments on MAWILAB datasets demonstrate that the proposed detection method outperforms other reference unsupervised detection methods. It achieves an accuracy of 99.86%, 99.97%, 97.08%, and 95.19% in overall four detection types including IP and port of source and destination.