Coordinated energy management for integrated energy system incorporating multiple flexibility measures of supply and demand sides: A deep reinforcement learning approach

With the development of energy Internet and intelligent buildings, the interactions of supply and demand sides of integrated energy system (IES) offer an attractive route for flexible energy management in buildings. However, traditional model-based control methods over-rely on precise mathematical modeling, difficult to flexibly deal with complex and changeable operating environments of IES. Therefore, this work proposes a coordinated operation optimization framework based on the deep reinforcement learning (DRL) algorithm for optimal scheduling of IES. Firstly, two supply-side and three demand-side flexibility measures are considered to tap the potential of flexible scheduling, including active adjustment of energy conversion equipment, energy storage, incentive-based demand response, electric vehicles and thermal inertia of building. Secondly, the coordinated optimization is formulated as a partially-observable Markov decision process. The twin delayed deep deterministic policy (TD3) algorithm is employed to solve the optimal energy management problem of IES, aiming at operation cost and user satisfaction. Thirdly, a hierarchical reward shaping (HRS) mechanism is proposed to improve the training performance of DRL, which could evaluate the current and final performance of agent and return the underway reward at each step and final reward. The developed optimization methodology is used for a case study in the building. The results show that the proposed HRS-TD3 algorithm achieves the fastest convergence and has the best economic performance compared with the other baseline algorithms. The operation cost of coordinated optimization is superior to those of the three baseline scenarios and achieves an improvement of 33.1%, 3.5% and 29.8%, respectively.