Goal-oriented Remote Tracking of an Unobservable Multi-State Markov Source

We study the problem of remote tracking in an energy-harvesting enabled status update system consisting of an information source, a sampler, a transmitter, and a monitor. The information source is modeled as a finite-state Markov chain. The sampler samples the source, and the transmitter transmits the taken samples to the monitor. We consider both sampling and transmission costs, and thus, the source is not fully observable. The primary objective is to determine the optimal joint sampling and transmission policies based on a goal-oriented metric, defined by a generic distortion function. We first formulate a stochastic optimization problem and cast it into a partially observable Markov decision process (POMDP) problem. Subsequently, we employ the notion of belief state and characterize the belief space through the age of information (AoI) to convert the problem into a finite-state MDP problem, which is then solved via the relative value iteration algorithm. We also explore different estimation strategies at the monitor and examine their impact on the system performance. The simulation results show the effectiveness of the derived policy and reveal that, depending on the source dynamic, the choice of estimation strategy itself can significantly influence the overall performance.