Augmenting Reservoirs with Higher Order Terms for Resource Efficient Learning

Reservoir computing provides an attractive alternative for time series data representation due to their gradient-free learning and energy efficient operation. In a ‘reservoir computer’ (RC), the reservoir is a random recurrent neural network that forms a high dimensional non-linear dynamical system which can be mapped to a desired sequence or dynamical system through linear regression. Although, the non-linearity of the reservoir gives its power, there is no guarantee that it is the most suitable for the task at hand. To this end, in this study, we propose to amplify the effectiveness of reservoirs by augmenting them with higher order terms computed based on reservoir unit activations. We name this model as Higher-order-augmented Reservoir Computer (ha-RC). We test the efficacy of ha-RCs by using two types of tasks: time series representation and long-term prediction, i.e. learning a dynamical system. Our experiments show that with the proposed model, a given learning accuracy can be achieved with significantly less memory resource compared to the baseline of standard Reservoir Computer (s-RC). This becomes possible as ha-RC can handle complex learning problems with much smaller reservoirs compared to s-RC models. This memory efficiency directly translates to energy efficiency as less number of arithmetic operations are needed with ha-RCs to reach the same level of accuracy compared to s-RCs. These points make our proposed model ideal for hardware implementation and edge computing.