Importance of Nyquist-Shannon Sampling in Training of Physics-Informed Neural Networks

Recent rapid advances in artificial intelligence methods and computational resources have opened up the possibility of solving complex problems in physics with machine learning. While promising, the training of performant deep learning models remains challenging, especially as the choice of various neural architectures or model hyper-parameters can all greatly impact model performance. Hence, it is now fairly common for researchers to systematically conduct either a neural architecture search (NAS) or hyper-parameter tuning as part of the model training process. However, these usually involve repeated model training and evaluation (typically via a cross-validation framework) and can be very computationally expensive. While there have been training-free heuristics proposed for NAS in recent years, this remains an active area of work, and much remains unresolved. Concurrently, physics-informed neural networks (PINNs) have been proposed in recent years as an attractive means of incorporating physics-derived knowledge into a deep learning framework for the modelling of real-world physical systems. However, recent literature has begun to emerge on the additional complexity of training PINNs due to the addition of these physics-derived equations (typically in the form of partial differential equations, PDEs). The extended optimization trajectories observed further exacerbate the aforementioned computational cost of NAS or hyper-parameter search. Hence, in this work, we propose a first demonstration of the utility of training-free heuristics for accelerating the selection of model parameters. In particular, we assess the correlation between basic concepts espoused in Nyquist-Shannon sampling, and the appropriate selection of mini-batch size for PINN training with stochastic gradient descent (specifically, the state-of-the-art ADAM algorithm). As a proof-of-concept, this paper focuses on training PINNs to predict the solution to a canonical 2D Helmholtz equation with varying complexity (as measured by the frequency of the signal present), and we evaluate and report the influence of different mini-batch size parameters on the effective training of PINNs. The results show that PINN model performance exhibits a step-like phase transition with increased mini-batch size during training. Critically, we note that the minimum mini-batch size used increases proportionally with the presence of higher frequencies in the physical system being modelled, as is intuitively described by the Nyquist-Shannon sampling theorem. This both shows the utility of similar potential heuristics for the training-free selection of hyper-parameters when training PINNs and suggests the potential for insights towards accelerated design and training of PINNs for physical systems from prior theoretical concepts in signal processing and reconstruction.