Multimodal Road Sign Interpretation for Autonomous Vehicles

Autonomous vehicles (AVs) are becoming increasingly prevalent. However, current AVs are unable to handle unexpected traffic signs (e.g., construction zones, road closures) encountered on the roads. To address this limitation, we propose MOSER, a Multimodal rOad Sign intERpretation system, to enable automated detection and interpretation of diverse road signs. Our system consists of a pipeline architecture with three main components, including perception, text processing, and planning. The perception component detects arbitrary road signs and extracts the sign text into proper groups and orders. The text processing component then identifies the high-level semantics of the text and determines whether any actions are required for the autonomous vehicle. Based on the interpretation of the signs, the planning component provides navigation guidance, such as instructing the vehicle to stop at a specific location or adding rules to its internal map. To the best of our knowledge, this is the first attempt to address the interpretation of arbitrary road signs using a multimodal processing strategy. Our work provides important insights and capabilities to support Level 4 autonomous vehicles, ensuring their safety and smoothness of operation.

I. Introduction

Within the last several years, autonomous vehicle (AV) technology has seen significant progress due to rapid advancements in computer vision [Sun+20]; [Liu+20]; [Vou+18] and artificial intelligence research [Wan+21]; [Bad+21]. Recently, AVs are being deployed in a growing number of major cities across the United States including San Francisco, Las Vegas, and Phoenix, with imminent plans to inaugurate driver-less taxis [NPR22]. However, one limitation of current AVs is that they often rely on pre-generated, high-definition maps – a data abstraction that captures "pre-recorded" information about an environment, which AVs use to navigate. Consequently, any acute changes to the environment can drastically affect their ability to operate, which can also have a significant and negative bearing on road safety.