The Improved Adaptive Algorithm of Deep Learning with Barzilai-Borwein Step Size

To solve the problem that it is difficult to determine the learning rate when training a neural network model, this paper proposes an improved adaptive algorithm based on the Barzilai-Borwein (BB) step size. In this paper, the new algorithm accelerates the model's training through the second-order momentum and adapts the learning rate according to the BB step size. We also set an adequate range for the learning rate to ensure the stability of adaptive adjustment and reduce the error of step size. Compared with different algorithms in a series of popular models, the new algorithm significantly avoids the tediousness of manually adjusting the learning rate and helps to improve the convergence speed. The results show that the new algorithm is feasible and effective.

I. Introduction

Deep learning has been successfully applied to virtual reality [1], [2]. It can accurately identify people's stat in the virtual environment by building a network model to identify their emotions [3] and postures [4], [5], and it can also be used for augmented reality [6], [7]. Deep learning needs to build various models for purpose and achieve different functions to output the desired results as much as possible. Moreover, the network models need to be optimized and trained to get a set of parameters to ensure the model outputs the desired results. Therefore, training a powerful network model for virtual reality requires an excellent optimization algorithm.