Anomaly Detection on Data Streams - A LSTM's Diary

In the past years, the importance of processing data streams increased with the emergence of new technologies and application domains. The Internet of Things provides many examples, in which processing and analyzing data streams are critical success factors. An important use case is to identify anomalies, i.e. something that is different or unexpected. We often have to cope with anomaly detection use cases in sequences within data streams, for instance in network intrusion, in predictive analytics or in forecasting. Sequence analysis can be performed using recurrent neural nets and in particular, we use long short-term memory (LSTM) neural nets. An LSTM is not only capable of storing a sequence of data but also of deciding to forget certain parts of it. Unfortunately, the internal representation of learned data does not clearly illustrate what was learned. Moreover, like many neural net-based approaches, these nets tend to need a high volume of data in order to produce valuable insights. In this paper, we want to present an experimental setting, comprising an architecture, a structured way of producing sample data and end-to-end pipelines to store and evaluate the hidden state of a LSTM per training batch. Main purpose is to extract the hidden state as well as to analyze changes during training and thus to identify patterns in the hidden state as well as anomalies.