Modeling, Optimization, and Operation of Large-Scale Air Traffic Flow Management on Spark

The nationwide air traffic flow management problem often encounters computational difficulty because it is generally modeled as an integer programming problem that requires computationally expensive optimization algorithms. This paper introduces a customized Spark-based optimization architecture for such large-scale integer programming problems to further speed up the modeling and optimization process, where Spark is a big data cluster-computing platform. First, a novel layered aggregate model is developed for handling flexible rerouting problem, which is not well handled in a previous link transmission model. As an aggregate linear model, the layered aggregate model has the nice features of computational efficiency and scalability, which make it suitable for Apache Spark. By applying a dual decomposition method, the original large-scale problem is decomposed into a number of small subproblems. The optimization proceeds by iteratively solving subproblems and updating Lagrange multipliers. This paper encapsulated the process into the Spark-based data processing model such that the optimization is automatically scheduled to run in parallel. Spark gains efficiency by means of in-memory computing and dynamic schedule allocation. This is demonstrated in the experimental results that are compared to an earlier Hadoop MapReduce-based model, where Hadoop MapReduce is a basic cloud computing framework; the Spark-based model is solved twice as fast as the Hadoop MapReduce-based model.