Remote Control of Bandits Over Queues - Relevance of Information Freshness

Sequential decision making for wireless networked control applications where the decision making agent is not colocated with the controlled system has been considered in prior work. The decision making agent obtains state observations and reward observations from the system through a communication network which may delay and lose such observations. Prior work has considered the relevance of information freshness for decision making agents which assume perfect knowledge of the system model. Motivated by the need to consider learning based decision making agents which have a fundamental exploration-exploitation tradeoff, we investigate the remote control of bandits over a bottlenecked communication network modelled as a Geo/Geo/1 queue. Using simulations, we investigate the regret performance of standard bandit algorithms such as
$\epsilon$
-greedy, upper confidence bound (UCB), and Thompson sampling in this context. We also propose reward-queue based bandit algorithms and evaluate their regret via simulations. We study the regret as a function of the queue arrival and service parameters as well as of the queueing discipline and investigate whether information freshness is relevant for remote learning based control of bandits.