Loading [MathJax]/extensions/MathMenu.js
Large Language Models for Human-Like Autonomous Driving: A Survey | IEEE Conference Publication | IEEE Xplore
Subscribe
ADVANCED SEARCH
Conferences >2024 IEEE 27th International ...

Large Language Models for Human-Like Autonomous Driving: A Survey

PDF
Yun Li; Kai Katsumata; Ehsan Javanmardi; Manabu Tsukada
All Authors

Abstract:

Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of ...Show More

Metadata

Abstract:

Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of Autonomous Driving (AD). As AD systems evolve from rule-based and optimization-based methods to learning-based techniques like deep reinforcement learning, they are now poised to embrace a third and more advanced category: knowledge-based AD empowered by LLMs. This shift promises to bring AD closer to human-like AD. However, integrating LLMs into AD systems poses challenges in real-time inference, safety assurance, and deployment costs. This survey provides a comprehensive and critical review of recent progress in leveraging LLMs for AD, focusing on their applications in modular AD pipelines and end-to-end AD systems. We highlight key advancements, identify pressing challenges, and propose promising research directions to bridge the gap between LLMs and AD, thereby facilitating the development of more human-like AD systems. The survey first introduces LLMs' key features and common training schemes, then delves into their applications in modular AD pipelines and end-to-end AD, respectively, followed by discussions on open challenges and future directions. Through this in-depth analysis, we aim to provide insights and inspiration for researchers and practitioners working at the intersection of AI and autonomous vehicles, ultimately contributing to safer, smarter, and more human-centric AD technologies.
Published in: 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)
Date of Conference: 24-27 September 2024
Date Added to IEEE Xplore: 20 March 2025
ISBN Information:

ISSN Information:

DOI: 10.1109/ITSC58415.2024.10919629
Conference Location: Edmonton, AB, Canada
References is not available for this document.
Contents

I. Introduction

Autonomous Driving (AD) has emerged as a transformative technology with the potential to revolutionize intelligent transportation systems, improve road safety, and enhance mobility. At the core of AD lies the decision-making process, which involves analyzing data, understanding the environ-ment, and making informed decisions about navigation and safety. As depicted in Fig. 1, the development of AD systems can be categorized into three categories based on the techniques employed.

Select All
1.
K. Yuan, Y. Huang, S. Yang, M. Wu, D. Cao, Q. Chen, et al., "Evolutionary Decision-Making and Planning for Autonomous Driving: A Hybrid Augmented Intelligence Framework", IEEE Trans. Intell. Transp. Syst, no. 99, pp. 1-13, 2023.
Google Scholar
2.
A. Aksjonov and V. Kyrki, "Rule-Based Decision-Making System for Autonomous Vehicles at Intersections with Mixed Traffic Environment", 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 660-666, Sep. 2021.
Google Scholar
3.
J. Guanetti, Y. Kim and F. Borrelli, "Control of connected and automated vehicles: State of the art and future challenges", Annu. Rev. Control, vol. 45, pp. 18-40, Jan. 2018.
CrossRef Google Scholar
4.
R. Zhao, Y. Li, F. Gao, Z. Gao and T. Zhang, "Multi-Agent Constrained Policy Optimization for Conflict - Free Management of Connected Autonomous Vehicles at Un signalized Intersections", IEEE Trans. Intell. Transp. Syst, no. 99, pp. 1-15.
View Article
Google Scholar
5.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van den Driessche, et al., "Mastering the game of Go with deep neural networks and tree search", Nature, vol. 529, no. 7587, pp. 484-489, Jan. 2016.
Google Scholar
6.
S. Aradi, "Survey of deep reinforcement learning for motion planning of autonomous vehicles", IEEE Trans. Intell. Transp. Syst, vol. 23, no. 2, pp. 740-759, Feb. 2022.
View Article
Google Scholar
7.
L. Wen, D. Fu, X. Li, X. Cai, T. Ma, P. Cai, et al., "DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models", International Conference on Learning Representations (ICLR 2024), 2024, [online] Available: http://arxiv.org/abs/2309.16292.
Google Scholar
8.
N. Kalra and S. M. Paddock, "Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?", Transp. Res. Part A: Policy Pract, vol. 94, pp. 182-193, Dec. 2016.
CrossRef Google Scholar
9.
S. Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, et al., "Dense reinforcement learning for safety validation of autonomous vehicles", Nature, vol. 615, no. 7953, pp. 620-627, Mar. 2023.
Google Scholar
10.
J. Chen, B. Yuan and M. Tomizuka, "Model-free Deep Reinforcement Learning for Urban Autonomous Driving", 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 2765-2771, Oct. 2019.
View Article
Google Scholar
11.
J. Lu, L. Han, Q. Wei, X. Wang, X. Dai and F.-Y. Wang, "Event-Triggered Deep Reinforcement Learning Using Parallel Control: A Case Study in Autonomous Driving", IEEE Transactions on Intelligent Vehicles, vol. 8, no. 4, pp. 2821-2831, Apr. 2023.
View Article
Google Scholar
12.
W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, et al., "End-to-end interpretable neural motion planner", 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8660-8669, Jun. 2019.
View Article
Google Scholar
13.
J. Kim and J. Canny, "Interpretable learning for self-driving cars by visualizing causal attention", 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2942-2950, Oct. 2017.
Google Scholar
14.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, et al., Sparks of Artificial General Intelligence: Early experiments with GPT-4, 2023, [online] Available: arXiv.org.
Google Scholar
15.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, et al., "Chain-of-thought prompting elicits reasoning in large language models", Advances in Neural Information Processing Systems, vol. 35, pp. 24824-24837, Jan. 2022.
Google Scholar
16.
J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, et al., "Finetuned Language Models are Zero-Shot Learners", International Conference on Learning Representations, Oct. 2022, [online] Available: https://openreview.net/forum?id=gEZrGCozdqR.
Google Scholar
17.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, et al., "Language models are few-shot learners", Adv. Neural Inf. Process. Syst, vol. 33, pp. 1877-1901, 2020.
Google Scholar
18.
R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, et al., "Imagebind: One embedding space to bind them all", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, vol. 15, pp. 180-15, 2023.
Google Scholar
19.
J. Li, D. Li, S. Savarese and S. Hoi, "BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models", International Conference on Machine Learning 2023, [online] Available: http://dx.doi.org/10.48550/ARXIV.2301.12597.
Google Scholar
20.
H. Liu, C. Li, Q. Wu and Y. J. Lee, Visual Instruction Tuning, Apr. 2023, [online] Available: http://arxiv.org/abs/2304.08485.
Google Scholar
21.
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, et al., "Flamingo: a Visual Language Model for Few-Shot Learning", 36th Confer-ence on Neural Information Processing Systems, [online] Available: https://proceedings.neurips.cc/paper_files/paper/2022/filel960a172bc7fbf0177ccccbb411a7d800-Paper-Conference.pdf.
Google Scholar
22.
A. Radford, K. Narasimhan, T. Salimans and I. Sutskever, Improving language understanding by generative pre-training, Jun. 2018, [online] Available: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
Google Scholar
23.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, et al., "LoRA: Low-Rank Adaptation of Large Language Models", International Conference on Learning Representations, Oct. 2021, [online] Available: https://openreview.net/forum?id=nZeVKeeFYf9.
Google Scholar
24.
G. Qin and J. Eisner, "Learning How to Ask: Querying LMs with Mixtures of Soft Prompts" in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association, pp. 5203-5212, Jun. 2021.
Google Scholar
25.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, et al., "Training language models to follow instructions with human feedback", Advances in Neural Information Processing Systems, vol. 35, pp. 27730-27744, Mar. 2022.
Google Scholar
26.
H. J. Jeon, S. Milli and A. Dragan, "Reward-rational (implicit) choice: A unifying formalism for reward learning", Conference on Neural Information Processing Systems, vol. abs/2002.04833, Feb. 2020, [online] Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/2flOc1578a0706e06b6d7db6fOb4a6af-Paper.pdf.
Google Scholar
27.
N. Wies, Y. Levine and A. Shashua, "The Learnability of In-Context Learning", Advances in Neural Information Processing Systems 36 (NeurIPS 2023), vol. 36, no. 15, pp. 36637-36651, Dec. 2023.
Google Scholar
28.
S. M. Xie, A. Raghunathan, P. Liang and T. Ma, "An Explanation of In-context Learning as Implicit Bayesian Inference", International Conference on Learning Representations, 2022, [online] Available: openreview.net.
Google Scholar
29.
M. Azarafza, M. Nayyeri, C. Steinmetz, S. Staab and A. Rettberg, Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving, Feb. 2024, [online] Available: http://arxiv.org/abs/2402.13602.
Google Scholar
30.
J. Liu, P. Hang, X. Qi, J. Wang and J. Sun, MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Un signalized Intersections, Jul. 2023, [online] Available: http://arxiv.org/abs/2307.16118.
Google Scholar

Contact IEEE to Subscribe
More Like This
Large Language Model for Medical Images: A Survey of Taxonomy, Systematic Review, and Future Trends

Big Data Mining and Analytics

Published: 2025

Navigating the Risks: A Review of Safety Issues in Large Language Models

2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)

Published: 2024

Show More

References

References is not available for this document.

IEEE Account

Purchase Details

Profile Information

Need Help?

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.
© Copyright 2025 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.