AggieGrid: from idle PCs to a distributed High-Throughput Computing system

For a long time scientists relied on large, powerful, and centralized systems, called High-Performance Computers, to run their computational jobs. Usually there was a long queue of jobs that needed to be finished before researchers could get their hands on the system. This entailed a lot of wait time and was not efficient enough for the users, so engineers and scientists started thinking of a different approach for computational research. As personal computers (PCs) became more popular, and every person at an institution had one, engineers and scientists came up with an idea to create decentralized systems from servers that were sitting in data centers, personal computers, and idle High-Performance Computing (HPC) resources. These systems enable users to utilize large amounts of computation power across the cluster and the era of decentralized High-Throughput Computing (HTC) began. We present and discuss our design, deployment, and administration of a decentralized HTC cluster called AggieGrid that was built from hundreds of idle student lab machines and ran successfully for over 4 years.