Comparative Study of Optimization Methods Proposed for the Extreme Learning Machine (ELM)

The Extreme Learning Machine (ELM) is a learning algorithm used for training a single hidden layer feed-forward neural network (SLFN). It leads to a better generalization performance and fast learning speed and many more advantageous properties. The first step of an ELM is to assign random values to the input weights and biases. Then it will determine the output weights in a single step using the generalized Moore-Penrose method. The random initialization of weights and biases and a large number of hidden nodes can affect the performance of the ELM. Several optimizers were proposed aiming to increase generalization performance and produce more compact networks. This paper presents a comparative study of ELM optimization in the state of the art using existing algorithms namely Optimally Pruned ELM (OP-ELM), Grey Wolf Optimizer (GWO), Salp Swarm Algorithm (SSA), Bat Algorithm (BA), and Particle Swarm Optimization (PSO) which are swarm intelligence-based ELMs. Also Genetic Pruning Algorithm (GPA) and Enhanced Genetic Algorithm (EGA) which are pruning methods using evolutionary algorithms. Results prove that SSA-ELM improves the accuracy in most benchmark datasets and shows better results than the other comparing methods while OP-ELM gives the most compact model with a high level of accuracy.

I. Introduction

Extreme Learning Machine [1] is a Single Hidden Layer Feed-forward Neural network (SLFN). It is characterized by its randomly assigned input weights and its analytically determined output weights using the Moore-Penrose generalized inverse. In order to provide a better generalization performance, which relies on the number of hidden nodes, pruning methods are used in the aim of reducing the size of models by removing irrelevant parameters then improving accuracy.