The Role of Regularization in Overparameterized Neural Networks

In this paper, we consider gradient descent on a regularized loss function for training an overparametrized neural network. We model the algorithm as an ODE and show how overparameterization and regularization work together to provide the right tradeoff between training and generalization errors.