Interpretation of Neural Network Players for a Generalized Divide the Dollar Game Using SHAP Values

Machine learning models can make accurate predictions but trust in the models depends on being able to under-stand why those predictions were made. Unfortunately, machine learning models are black boxes making interpretation difficult. Previously we used an evolutionary algorithm to evolve triplets of neural network players for instances of the Generalized Divide-the-Dollar, which is an economic bargaining game. The players produced fair bids with high bid totals, which is a desirable outcome, but no attempt was made to understand why the players performed so well. In this paper, we interpret the behavior of those neural networks using SHapley Additive exPlanations (or SHAP). Surprisingly, the neural network players exhibited both altruistic and exploitative behavior. Both a global and a local interpretation analysis is presented. The experiments conducted in this work demonstrate a simple method for understanding players' strategies in multi-player gamcs.