Energy-efficient sorting using solid state disks

The energy efficiency of data processing is becoming more and more important for both economical and ecological reasons. In this paper, we take sorting of large data sets as a representative case study for data-intensive applications. Guided by theoretical algorithmic considerations and taking practical limitations into account, we carefully choose the components for building an energy-efficient computer for this task. These decisions are backed up by performance and power measurements of two competing options. Finally, we choose a low-power Intel Atom 330 processor, supported by four solid state disks, which have little power consumption and provide high bandwidths. Our sophisticated implementation of the sorting algorithms does not only feature great CPU efficiency. By employing overlapping, it loads all available resources in parallel, resulting in a good overall balance between I/O and computation. Using this setup, we beat the former records in the JouleSort category of the well-established Sort Benchmark for inputs from 10 GB to 1 TB of data, by factors of up to 5.1. This usually comes without a penalty in running time. We break another JouleSort record using a standard server machine, which showcases the general energy efficiency improvements in standard hardware over the years. Furthermore, we present the first-ever result in the 100 TB JouleSort category, on a large compute cluster. The results lead us to conclusions on how to design scalable energy-efficient systems for processing large data sets, such as combining relatively weak computing power with high bandwidth storage devices. We also speculate on the consequences of future hardware for the Sort Benchmark contest, and identify certain problems, also relating to the monetary cost of energy. (C) 2011 Elsevier Inc. All rights reserved.