Slim LSTM NETWORKS: LSTM_6 and LSTM_C6

We show that parameter-reduced slim variants of Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNN) are comparable in performance to the standard LSTM RNN on two diverse benchmark datasets, namely, the review sentiment IMBD and the 20 Newsgroup datasets. We focus on two of the simplest slim variants, namely LSTM_6 (i.e., standard LSTM with three fixed constant gates) and LSTM_C6_(i.e., LSTM_6 with further reduced memory-cell input block). We demonstrate that these two aggressively reduced-parameter variants are competitive with the standard LSTM when hyper-parameters, e.g., the learning rate, number of hidden units and/or gate constants are set properly. These slim architectures enable training speedup and hence these networks would be more suitable for training and inference onto portable devices with relatively limited computational resources.