Optimal Control of Discrete-Time Multivariate Point Processes with Finite-Time Steady State

Multivariate point processes (MPP) are widely used to model the occurrence of multiple interrelated events in complex systems. They are used in a variety of fields to analyze data and define models that can make predictions about future events. In this paper, we consider finite-state MPP, which are products of finite state automata (FSA) and MPP. Specifically, the events in the MPP trigger state transitions in the FSA, while the intensities of the point processes are defined as functions of the FSA state and the history of the MPP. Further, we assume that some of the event types are controllable, i.e., they are not random but can be triggered. We formulate an optimal control problem for such system, which can then be expressed as optimal control for a Markov Decision Process (MDP) with infinite states. When the system has appropriate finite-time steady state properties, we use the concept of stochastic bisimulation of MDP to reduce the MDP into a finite state one, thereby allowing us to use standard optimal control techniques to calculate the optimal policy. We demonstrate the effectiveness of our method on a simplified sleep-wake cycle model, for the problem of optimally scheduling naps to maximize the length of wakefulness intervals.