Service Function Chaining in LEO Satellite Networks via Multi-Agent Reinforcement Learning

Low-earth-orbit satellite networks (LSNs) offer an enhanced global connectivity and a wide range of applications such as disaster response and military operations, among others. Each specific application can be represented by a service function chain (SFC) in which each function is considered as a task in the application. Our objective is to optimize the long-term system performance by minimizing the average end-toend delay of SFC deployments in LSNs. To achieve this, we formulate a dynamic programming (DP) problem to derive an optimal placement policy. To overcome the computational intractability, the need for statistical knowledge of SFC requests, and centralized decision-making challenges, we present amulti-agent Q-learning approach where satellites act as independent agents. To facilitate performance convergence in non-stationary agents' environments, we let agents to collaborate by sharing designated learning parameters. In addition, agents update their Q-tables via two distinct rules depending on selected actions. Extensive experimentation shows that our approach achieves convergence and performance relatively close to the optimum obtained by solving the formulated DP equation.