Data Partitioning Method for Efficient Parallel Skyline Computation

Skyline computation has long been an important and hot research topic in the field of data management. Given a set of multi-dimensional data points, a Skyline operator selects the points that are not dominated by any other points on all dimensions; the process of processing Skyline operator is referred as Skyline computation. Skyline operator enables users to select objects of interest from a comparatively small set of Skyline result set, without the need of care about those objects that have been filtered out. As a consequence, Skyline computation finds various applications like multi-criteria decision-making, visual data analytics, and user preference search, etc., and typical application tasks include but not limited to business marketing strategy analysis, production capability lateral assessment, etc. With the arrival of big data era, as well as the wide application of distributed networks and the rapid development of cloud computing platform-based solutions, the increase of data volume in various application domains has become a key technical challenge, and as a consequence, parallel Skyline operator for large-scale data sets was proposed, in order to partially resolve the difficulty imposed by big data; meanwhile, related research on parallel Skyline computation has received amplified attention from both academia and industry lately. Due to the lack of global distributional information regarding the overall dataset, processing of parallel Skyline computation is facing great technical challenges. Generally speaking, the computational framework of parallel Skyline processing normally comprises three major steps: (1)appropriately partition the possibly big dataset; (2)using local computing resources to evaluate local Skyline on each partition respectively; and (3) merge local Skylines into eventually a global Skyline. Among them, there are many existing algorithms for the latter two steps, i.e., compute local Skylines and merge local Skylines, and moreover, the related research is comparatively mature; on the contrary, related research on the first step is comparatively rare, although the effect of the first step directly determines the parallelism of the entire computation, and hence, the overall performance of the parallel computing system. Specifically, for the first step, there are two criteria that need to be considered: (1)whether computation workloads are balanced across partitions; (2)how to reduce local Skyline cardinalities on each partition. However, existing parallel Skyline algorithms, no matter random partitioning or grid-based method, can only satisfy only one of the criteria but not both. To address the issue, we exploit a probabilistic model to evaluate Skyline cardinality, and the probabilistic model can encapsulate relevant results from existing literature into a unified framework. Then, based on that, propose a novel permutation-based method for data partitioning which through simple data point mapping is able to achieve both load balance and generate smaller Skyline candidate sets in comparison with other existing methods. On the solid basis of theoretical study, extensive experiments on large-scale synthetic data sets and real-life data sets verify the effectiveness of the proposed model as well as the method; in other words, in comprehensive experimental study, the proposed method improves the execution efficiency of parallel Skyline operator, and it outperforms existing algorithms alike under most parameter settings. © 2020, Science Press. All right reserved.