A Q-Learning Based Transmission Management Strategy for Energy Harvesting Sensors

Radio-frequency energy harvesting from ambient cellular energy and drone-based receivers close to the sensors can be effective tools to prolong the lifespan of wireless sensor networks (WSNs). Moreover, the energy management policy of a sensor plays a critical role in increasing the reliability of data transmission under severe energy constraints. Thus, in this paper, we develop an optimal transmission policy to reduce the outage such that a sensor decides on when to transmit and how much power to use given the uncertainties of the harvested energy. To this end, we model sensors with a finite-level battery and a buffer with discrete states, whereas the cellular base stations from which the energy is harvested are modeled according to a Poisson point process. The drone is assumed to be hovering at a fixed height above the field of sensors, and assume path loss and small scale fading with varying intensities depending on the line-of-sight nature of the link. An outage is assumed to occur due to incorrect reception of transmissions and buffer overflow. We formulate the problem as a Markov decision process, and utilize a Q-learning based algorithm to generate the optimal transmission policy. Our numerical results show that the proposed policy significantly outperforms the previously proposed policies under all conditions.