Bridging MQTT and Kafka to support C-ITS: a feasibility study

Nowadays, every vehicle is wirelessly connected, often by both itself and its passengers. The same goes for all the continuously growing number of devices alongside the transportation infrastructure. This induces an abundance of data that can be used to direct traffic efficiently, optimize the transportation infrastructure, avoid accidents, lower emissions from transportation and so forth. To create coherence among the messages related to a Cooperative and Intelligent Transportation System (C-ITS), an ETSI message type standard has been developed. The next hurdle is how to facilitate receiving and processing millions of C-ITS messages in real-time. We present a feasibility study on how these C-ITS messages can be received, processed and distributed. Our novel approach is to bridge C-ITS message types over MQTT, to Apache Kafka, with fault-tolerance, horizontal scalability, and low latency. We have engineered both an MQTT-Kafka source connector and a Kafka-MQTT sink connector, by utilizing the Kafka Connector API. These facilitate an easy configurable topic mapping between MQTT and Kafka, from vehicles and roadside units to a central application, and in the opposite direction. Our experiments with this bridging technology, performed on a modest desktop computer with a single instance Kafka setup, show the feasibility by managing to source and sink CAM messages with low latency.