Elastic Partition Placement for Non-stationary Graph Algorithms

Distributed graph platforms like Pregel have used vertex-centric programming models to process the growing corpus of graph datasets using commodity clusters. However, the irregular structure of graphs causes load imbalances across machines, and this is exacerbated for non-stationary graph algorithms where not all parts of the graph are active at the same time. As a result, such graph platforms do not make efficient use of distributed resources. In this paper, we decouple graph partitioning from placement on hosts, and introduce strategies for elastic placement of graph partitions on Cloud VMs to reduce the cost of execution compared to a static placement, even as we minimize the increase in makespan. These strategies are innovative in modeling the graph algorithm's non-stationary behavior a priori using a metagraph sketch. We validate our strategies for several real-world graphs, using runtime traces for approximate Betweenness Centrality (BC) algorithm on our subgraph-centric GoFFish graph platform. Our strategies are able to reduce the cost of execution by up to 54%, compared to a static placement, while achieving a makespan that is within 25% of the optimal.