A Distributed Policy Evaluation Scheme Over Consensus Iteration

This paper develops a distributed policy evaluation scheme under the scenario that a group of agents collaborate to estimate the value function of a given policy with the global state and the local reward. Under this framework, we formulate a cost function as a combination of linear function approximation and eligibility traces. Combining gradient decent and consensus iteration with time-averaging, a distributed policy evaluation algorithm is proposed and the convergence is shown by appropriately setting the gradient learning rates and the numbers of consensus iterations. The proposed policy evaluation scheme has the potential to improve the robust performance in the learning process. Experimental results validate the theoretical analyses of our algorithm.