Partially Observable Markov Decision Subject to Total Variation Distance Ambiguity

In this paper we consider such Markov decision problems on finite spaces, when the probability distribution of the output conditioned on the sufficient statistic, belongs to a ball, with respect to the total variation distance metric, centered around a nominal conditional distribution. We formulate the Markov decision problem, as a minimax optimization problem in which the minimization is over the admissible set of control strategies, while the maximization is over the conditional distributions which lie in the total variation distance ball. Then we present a two-step procedure to determine the optimal control strategies. In step 1, we determine the maximizing conditional distribution, as a function of the nominal conditional distribution and the radius of the ball; it is defined on a finite partition of the observation space of aggregating the output states to form new states. In step 2, the minimax problem is transformed into a minimization problem over the control strategies, and a new dynamic programming equation is derived, where the conditional expectation is taken with respect to the maximizing conditional distribution. Surprisingly, although the current paper deals with partially observable Markov decision problems, the mathematical analysis is tractable.