Advances in Morphological Neural Networks: Training, Pruning and Enforcing Shape Constraints

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphological classifier as a Difference-of-Convex optimization problem and extend this method to multiclass tasks. We then focus on general morphological networks trained with gradient descent variants and show, quantitatively via pruning schemes as well as qualitatively, the sparsity of the resulted representations compared to FeedForward networks with ReLU activations as well as the effect the training optimizer has on such compression techniques. Finally, we show how morphological networks can be employed to guarantee monotonicity and present a softened version of a known architecture, based on Maslov Dequantization, which alleviates issues of gradient propagation associated with its "hard" counterparts and moderately improves performance.

1. INTRODUCTION

During the last decade, Neural Networks have been the focal point of machine learning research, especially in the dawn of the Deep Learning era. Most architectures utilize the multiply-accumulate scheme of the linear perceptron that feeds into a nonlinearity. An alternative approach lies on the use of morphological neurons, first introduced by Davidson and Hummer [1]. This approach was extended by Ritter and Sussner, where a simple network consisting of a single hidden layer was proposed for binary classification tasks resulting in a decision boundary parallel to the axes [2]. This limitation was addressed in two major ways, either by extending the architecture to a second hidden layer, where numerous such hyperplanes can be learned allowing the solution of arbitrary (binary) classification tasks [3] or by adding the option of hyperplane rotation [4].