Incremental Cluster Validity Index for Predicting Early Signs of Change in Data Streams

In this paper, we propose an incremental version of the SD cluster validity index for streaming data monitoring and predicting early signs of change. The proposed incremental SD (iSD) is used to monitor the data stream along with the MU Streaming Clustering (MUSC) algorithm. We investigate the use of iSD for detecting early signs of changes in multiple data streams arriving at the same time. Synthetic and real-life datasets are used in the analysis to demonstrate the effectiveness of the proposed index in detecting early signs of changes in the data stream. Valuable information about the streaming data can be directly captured from the index values such as the appearance of new patterns, and cluster size based on the analysis of outliers. The performance of iSD is compared with the incremental Davies-Boudin index (iDB). iSD has larger values which makes it more robust in monitoring large data streams compared to iDB which tends to flatten over time and approaches zero.