Learning in sinusoidal spaces with physics-informed neural networks

A physics-informed neural network (PINN) uses physics-Augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this article, we present a novel perspective of the merits of learning in sinusoidal spaces with PINNs. By analyzing behavior at model initialization, we first show that a PINN of increasing expressiveness induces an initial bias around flat output functions. Notably, this initial solution can be very close to satisfying many physics PDEs, i.e., falling into a localminimum of the PINN loss that onlyminimizes PDE residuals, while still being far from the true solution that jointly minimizes PDE residuals and the initial and/or boundary conditions. It is difficult for gradient descent optimization to escape from such a local minimum trap, often causing the training to stall. We then prove that the sinusoidalmapping of inputs-in an architecture we label as sf-PINN-is effective to increase input gradient variability, thus avoiding being trapped in such deceptive local minimum. The level of variability can be effectively modulated to match high-frequency patterns in the problem at hand. A key facet of this article is the comprehensive empirical study that demonstrates the efficacy of learning in sinusoidal spaces with PINNs for a wide range of forward and inversemodeling problems spanning multiple physics domains. Impact Statement-Falling under the emerging field of physicsinformed machine learning, PINN models have tremendous potential as a unifying AI framework for assimilating physics theory and measurement data. However, they remain infeasible for broad science and engineering applications due to computational cost and training challenges, especially for more complex problems. Instead of focusing on empirical demonstration of applicability to a specific problem, this paper analyzed and provided novel perspectives on why a typical PINN implementation frequently exhibits training difficulty, even on a seemingly straightforward 1D model problem. Interestingly, the issues uncovered explain how common heuristics for data-fit neural networks (e.g. initialization) cause training difficulties in this new paradigm of PINNs. Critically, we believe this to be among the first work to analytically provide insight into how simply adapting conventional neural network implementations may be broadly problematic for PINNs across different PDEs (physics). Guided by this insight, we then further provide theoretical results on the benefit of learning in sinusoidal space with PINNs. Utilizing sinusoidal spaces initializes sf-PINNs with input gradient distributions that can overcome the trainability issues uncovered, and this is further demonstrated by a comprehensive empirical study for a wide range of forward and inverse modelling problems spanning multiple physics domains. sf-PINNs can significantly improve the model accuracy by a few orders of magnitudes. Our paper also provides a theoretical foundation for unifying and explaining purported benefits from previously published empirical works, thereby advancing our understanding of this new PINN paradigm. © 2024 Institute of Electrical and Electronics Engineers Inc.. All rights reserved.