Investigating Echo State Network Performance with Biologically-Inspired Hierarchical Network Structure

The nervous systems of animals exhibit complex recurrent topologies. These networks often show key properties like small-worldness and clustering, forming a complex hierarchical topology that is very different from many standard random graphs. Current machine learning techniques are dominated by feed-forward networks without recurrent connectivity, or specialized architectures for gated recurrent connectivity (such as Long-Short Term Memory Networks). Reservoir computing networks, such as echo state networks, represent an exciting alternative form of recurrent networks, which require limited training data compared to other recurrent networks. Echo state networks, however, typically use randomized network topologies. We introduce a novel reservoir computing network, with a hierarchical network structure inspired by organization of biological networks, utilizing hierarchical stochastic block models. We demonstrate the use of this network for predicting dynamic system evolution, and we compare this network to existing echo state network topologies. We find that the best performing networks utilize our new topology, and, for Mackey-Glass system prediction as well as MNIST classification, the overall distribution of solutions is improved. For the Mackey-Glass system, the median population NRMSE improved from 0.856 to 0.08, and for MNIST the median population accuracy increased from 92.5% to 96.8%. This may be preliminary indication of improving network function through changing network topology. The spectral radii of the produced HSBM topologies do not always obey the echo state property, and we show that both the spectral radius as well as common graph metrics do not predict the best performing networks. This argues for further investigation into the structural properties of high performing networks utilizing reservoir computing.