Two-Stage Genetic Algorithm for Designing Long Short Term Memory (LSTM) Ensembles

Long Short Term Memory (LSTM) is a special kind of Recurrent Neural Networks popularly used in various applications. However, using a single LSTM is often not enough to attain reliable performance on complicated machine learning tasks. This is because LSTM is sensitive to the specifics of the training data. Ensemble learning is a promising approach to improve the performance of LSTMs on complicated tasks. However, it is difficult to design an ensemble of LSTMs. LSTMs that constitute the ensemble should be both accurate and diverse. This paper proposes a new two-phase evolutionary algorithm to design ensembles. The first phase is to evolve best performing LSTMs automatically. A connection weight inheritance approach is used in the first phase to improve the effectiveness and efficiency of the evolutionary process. The second phase is to design ensembles by choosing suitable LSTMs without fixing the ensemble size in advance. We use bagging to train the selected LSTMs to build the ensemble to achieve good diversity among the LSTMs. The proposed approach is evaluated on various classification tasks. The results show the effectiveness of the proposed approach and its significant improvement in performance over many state-of-the-art machine learning models. The results also show the efficiency of the proposed approach in comparison with the baseline algorithm.