Dynamic Data Streaming for an Appliance

Many applications require to analyse large amounts of continuous flows of data produced by different data sources before the data is stored. Data streaming engines emerged as a solution for processing data on the fly. At the same time, computer architectures have evolved to systems with several interconnected CPUs and Non Uniform Memory Access (NUMA), where the cost of accessing memory from a core depends on how CPUs are interconnected. In order to get better resource utilization and adaptiveness to the load dynamic migration of queries must be available in data streaming engines. Moreover, data streaming applications require high availability so that failures do not cause service interruption and losing data. This paper presents the dynamic migration and fault-tolerance capabilities of UPM-CEP, a data streaming engine designed to take advantage of NUMA architectures. The preliminary evaluation using Intel HiBench benchmark shows the effect of the query migration and fault-tolerance on the system performance.