Data assimilation for urban stormwater and water quality simulations using deep reinforcement learning

Hydrological models have been used to understand the transportation of water quantity and quality in drainage systems, and the stormwater management model (SWMM) is one of the most widely-used models for runoff quantity and quality simulations in urban areas. Although significant efforts have been made to identify the appropriate input parameters of the SWMM model in various watersheds, it is difficult to reflect the variability of the real environment using a fixed input parameter. Data assimilation (DA) is a compatibility method that improves prediction accuracy by merging observations and simulation outputs. It is useful to reduce the forecast error from temporal transferability with information interaction between the model output and observation data. However, conventional DA approaches cannot completely overcome their unrealistic assumptions such as linearity, normality, and error covariances. To address these challenges, we used deep reinforcement learning (RL) to develop an automatic assimilation model that interactively optimizes the SWMM parameters in a real environment (SWMM-RL). In the SWMM-RL model, the agent is trained by rewarding and/or punishing the action (modulated SWMM input parameter) in real time according to the state changes. The model was constructed to minimize the error of runoff and pollutant load simulation of suspended solids (SS), total nitrogen (TN), and total phosphorus (TP) in each stormwater monitoring event. The results demonstrated that the SWMM-RL model primarily outperformed the fixed input parameter SWMM model (SWMM-PS) in simulating the runoff and the three different pollutant loads (the median value of Nash-Sutcliffe efficiency (NSE) for 10 stormwater events was increased by 0.11, 0.31, 0.36, and 0.07 for runoff, SS, TN, and TP, respectively). The SWMM-PS has disadvantages in simulating low rainfall events because of the high sensitivity of runoff peaks to heavy rainfall conditions. Furthermore, the effects of extreme rainfall events were estimated using the SWMM-RL model. This study showed how SWMM-RL combined with the DA method increased forecast accuracy by providing sensitivity and temporal transitions of input parameters.