Parallel Job Processing Technique for Real-time Big-Data Processing Framework

Since the introduction of big data, numerous researches aiming to improve the accuracy and speed of data processing has been conducted. Many platforms that can process real-time data were developed for this purpose. Most standard data processing platforms used Spark Streaming as data analysis layer. However, its limitation in performance calls for a better alternative. This paper introduces a new data processing framework, Squall. Squall utilizes parallel processing and allows real-time data processing using streaming modules. Go was used for development. Through various experiments, the performance of our newly developed framework on processing real-time data was compared to the performance of the previously existing framework completing the same task. Results show quantitative evidence that Squall excel the platforms that use Spark Streaming. Our future work includes making modifications that will improve Squall's performance.