Relating Bayesian learning to training in recurrent networks

It is demonstrated that a recurrent neural network relying on an error correcting learning algorithm and a localist coding scheme is able to converge to a solution that would be expected from Bayesian learning. This is possible even without implementing Bayes theorem and without assigning prior probabilities to the model.

I. Introduction

Machine Learning and Statistics have many aspects in common. It will be shown that Bayes theorem can contribute to understand and predict to what solution recurrent networks with truncated gradient descent learning algorithms and a localist coding scheme converge. I have chosen a simple recurrent network, SRN [1], for my demonstration.