HybridTE2: Hybrid Transformer-based End-to-End Learning for Autonomous Driving

Accurate navigation prediction is paramount for autonomous driving, offering the potential to enhance safety and efficiency by mitigating accidents caused by human error. This task necessitates the capture of long-term dependencies and complex correlations, especially in challenging driving scenarios. While Convolutional Neural Networks (CNNs) have historically been employed to predict navigation commands, delivering good accuracy, they often struggle in capturing long-term dependencies. In contrast, attention-based models like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks exhibit proficiency in capturing long-term dependencies. This research is driven by a fundamental question: Which types of architectural designs are best suited to effectively capture these long-term dependencies in the context of precise navigation prediction. To tackle the question, a hybrid Transformer-based end-to-end architecture, HybridTE
²
, was designed for predicting steering angles and speed in autonomous driving scenarios. The proposed architecture integrates the strengths of CNNs and Transformer Encoder, leveraging CNNs to efficiently extract lowlevel features while simultaneously harnessing the Transformer’s ability to capture long-term dependencies. To empirically validate the proposed architecture, our evaluation has been conducted using the real-world large-scale dataset DBNet for driving behavior learning. The experimental results demonstrate that the proposed HybridTE
²
architecture significantly outperforms the state-of-the-art CNN-based Nvidia’s PilotNet architecture and other baseline architectures.