Neural Network Optimization with Biologically Inspired Low-Dimensional Manifold Learning

Neural Networks learn to recognize and leverage patterns in data. In most cases, while data is represented in a high-dimensional space, the patterns within the data exist along a manifold in a small subset of those dimensions. In this paper, we show that by using a biologically inspired algorithm called Geometric Multi-Resolution Analysis (GMRA), these low-dimensional manifolds can be computed and can be used to convert datasets into more useful forms for learning. We also show that, thanks to the lower-dimensional representation of the converted datasets, that smaller networks can achieve state-of-the-art performance while using significantly fewer parameters.