Single Agent Formulation for Reinforcement Learning Based Routing of Urban Last Mile Logistics with Platooning Vehicles

Last mile logistics are in the midst of a deep transformation thanks to the advent of autonomous vehicles with platooning capabilities that can take the place of typical delivery methods. Platooning brings to the vehicle routing problems new constraints and multiple objectives that are addressed in this paper with a Reinforcement Learning approach. In opposition to traditional metaheuristic optimization algorithms, Reinforcement Learning provides flexibility in the face of changing environment, shifting the challenge to the way in which the problem is formulated. While there have been successful attempts to implement RL solutions to vehicle routing problems, including some sort of optional platooning, our main contribution is funded in the application to this platooning vehicle routing problems for last mile delivery, considering all their particularities and proposing a formulation framework for this kind of problems.

I. Introduction

The demand for swift and efficient last - mile logistics has soared in recent years, placing immense strain on the final segment of supply chains. This demand has been addressed by logistic service providers (LSPs) with the expansion of last mile delivery fleets, with its subsequent impact on traffic congestion, air pollution, and operational inefficiencies, ultimately diminishing the urban quality of life [1]. Several alternative last mile logistic strategies have been devised from industry and academia to alleviate the impact of this growth. Research in this field ranges from distributing micro delivery hubs within the urban network, to changing the composition and behavior of the delivery fleet. Thus, electric vehicles or cargo bikes can alleviate pollution, while unmanned vehicles, ground or aerial, can improve congestion issues. This kind of vehicles can operate isolated or in platoons[2], and in in this last case, delivery vehicles navigate the road network in platoons, offering extra advantages such as reduced energy consumption due to decreased drag, minimized congestion through unified operation, and the flexibility to add or detach vehicles from the platoon [3]. However, implementing such delivery strategies poses various challenges, prompting extensive research efforts that regard considerations like communication, perception, and control [4], [5].