A Fitness Landscape Analysis of the LeNet-5 Loss Function

Fitness landscape analysis (FLA) refers to a set of techniques that allow for the characterisation, visualisation and comprehension of the trends of objective functions within their decision spaces. Two of the important features estimated through FLA are ruggedness, i.e. the number and distribution of optima within the decision space, and neutrality, i.e. the width, distribution and frequency of areas with one objective function value. Recent studies have proposed the application of FLA to the training problem in supervised learning. The present paper extends previous results on ruggedness and neutrality to investigate the loss landscape of LeNet-5. More specifically, we demonstrate that the neutrality threshold is an important metaparameter and that its optimisation is required to correctly assess the ruggedness and neutrality of a problem. Furthermore, this study investigates the difference between the landscape perceived by a learning algorithm and the actual landscape. The obtained numerical results indicate that the division of datasets into batches causes an increase in perceived ruggedness. As a result, modern training algorithms that make use of division into batches, such as Adam, overestimate ruggedness.