MARL for Autonomous Driving

Hi there, welcome to our repository.
Here is all you need to learn the Multi-Agent Reinforcement Learning (MARL) for Autonomous Driving.
For a more comprehensive survey, please look at:

Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey
Ruiqi Zhang^1,2, Jing Hou¹, Florian Walter³, Shangding Gu^2,4, Jiayi Guan¹, Florian Röhrbein⁵, Yali Du⁶, Panpan Cai⁷, Guang Chen^1,4,*, Alois Knoll^4
1Tongji University; ²UC Berkeley; ³University of Technology Nuremberg; ⁴TUM; ⁵Chemnitz University of Technology; ⁶KCL; ⁷SJTU

0 Glance at the History

We have summarized the following over the past decades: (1) autonomous driving simulators, datasets, and competitions; (2) development trends in hardware and software; (3) single-agent and multi-agent reinforcement learning and related algorithms.

1 Open-Access Learning Materials

Books

Reinforcement Learning: An Introduction (by Richard Sutton et. al, MIT press)
Reinforcement Learning for Sequential Decision and Optimal Control (by Shengbo Eben Li, Springer)
Autonomous Driving (Technical, Legal and Social Aspects) (by Markus Maurer et. al, Springer)

Courses

UC Berkeley, CS285: Introduction to Reinforcement Learning (by Sergey Levine)
Stanford, CS 234: Reinforcement Learning (by Emma Brunskill)
Introduction to Reinforcement Learning (Chinese Version) (by Bolei Zhou)
Multi-Agent Artificial Intelligence [Bilibili] (by Jun Wang)
Self-Driving Cars [Course Syllabus] [Video] (by Andreas Geiger)

Talks

MicroSoft Reinforcement Learning Day: Multi-Agent Reinforcement Learning
Berkeley Simons Institute: Multi-Agent Reinforcement Learning [Part I][Part II]
Safe Reinforcement Learning via Statistical Model Predictive Shielding, RSS 2021
Safety in Reinforcement Learning by Leveraging Offline Data, IEEE MFI 2022
Learning Robust Policies for Self-Driving, ECCV 2022

2 Benchmarks

2.1 Simulators

TrafficFlow Oriented

Simulator	Released Time	Paper	Other Supplyments	Affiliation
SUMO	2001	Preprint	-	openMobility
Flow	2018	T-RO	Documentation	UC Berkeley
Highway-env	2018	-	Documentation	Farama FD.
CityFlow	2019	WWW	Documentation	UC Berkeley
BARK	2020	IROS	Documentation	fortiss
MADRaS	2020	-	Documentation	-
SMARTS	2020	CoRL	Documentation	Noah's Ark Lab
MetaDrive	2021	T-PAMI	Documentation	UCLA
TBSim	2021	ICRA	Pretrained Model	NVIDIA Research
TorchDriveSim	2021	ITSC	-	Inverted AI
InterSim	2022	IROS	-	Tsinghua University
Nocturne	2022	NeurIPS	-	Meta
ScenarioNet	2024	NeurIPS	Documentation	UCLA
Waymax	2024	NeurIPS	Documentation	Waymo Research

Fidelity Oriented

Simulator	Released Time	Paper	Other Supplyments	Affiliation
TORCS	2000	-	-	SourceForge
Gym-TORCS	2017	ArXiv	-	UTokyo
CARLA	2017	CoRL	Documentation	Intel Lab
MACAD	2020	IJCNN	-	MicroSoft Research
ISAAC Sim	2020	-	Documentation	NVIDIA Research
Vista	2020	RA-L /ICRA	Documentation	MIT CSAIL
ISAAC Lab	2024	RA-L	Documentation	NVIDIA Research

2.2 Datasets

Simulator	Released Time	Affiliation
KITTI	2013	KIT
Visual KITTI	2016	Naver Lab
INTERACTION Dataset	2019	UC Berkeley
Visual KITTI 2	2020	Naver Lab
KITTI 360	2021	KIT
nuScenes	-	Motional
nuPlan	-	Motional
Waymo Open Dataset	-	Waymo
Lyft LV5	-	Lyft

3. Methodologies

3.1 Fundamental Algorithm

Model-Free RL

Algorithm	Released Time	Paper	Implementation	Affiliation
Deep Q-Network	2013	Preprint	SB3, Official	DeepMind
DDPG	2015	ICML	SB3	DeepMind
Double DQN	2015	AAAI	SB3	DeepMind
Dueling DQN	2016	ArXiv	SB3	DeepMind
REINFORCE	1992	Machine Learning	Official	Northeastern University
TRPO	2015	ICML	SpinUp	UC Berkeley
A2C	2016	ICML	SB3	DeepMind
PPO	2017	ArXiv	SB3	OpenAI
TD3	2018	ICML	SB3	McGill University
SAC	2018	ICML	SB3	UC Berkeley

Model-based RL

Algorithm	Released Time	Paper	Implementation	Affiliation
MBPO	2019	NeurIPS	Official	UC Berkeley
PlaNet	2019	ICML	Official	Google
Dreamer v1	2020	ICLR	Official	Google
Dreamer v2	2021	ICLR	Official	Google
Dreamer v3	2023	ArXiv	Official	DeepMind

3.2 Multi-Agent Reinforcement Learning (MARL) Algorithms

Algorithm	Released Time	Paper	Implementation	Affiliation
IQL	2015	ArXiv	Pymarl	University of Tartu
VDN	2017	ArXiv	Pymarl	DeepMind
MADDPG	2017	NeurIPS	Official	OpenAI, UC Berkeley
COMA	2017	AAAI	Pymarl	University of Oxford
QMIX	2018	ICML	Pymarl	University of Oxford
QTRAN	2019	ICML	Pymarl	KAIST
IPPO	2019	ArXiv	epymarl	University of Oxford
AlphaStar	2019	Nature	Official	DeepMind
MAPPO	2021	NeurIPS	Official	Tsinghua, UC Berkeley

---

UPDATE IS STILL ON THE WAY (after Sep.15, 2024)

3.3 CTDE MARL for Autonomous Driving

3.4 Decentralied MARL for Autonomous Driving

3.5 MARL with Social Preference

3.6 Trust-worthy and Safe MARL

Citation

If this repository or our paper is useful for your research and would like to cite it, here is our bibtex.

@article{zhang2024multi,
  title={Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey},
  author={Zhang, Ruiqi and Hou, Jing and Walter, Florian and Gu, Shangding and Guan, Jiayi and R{\"o}hrbein, Florian and Du, Yali and Cai, Panpan and Chen, Guang and Knoll, Alois},
  journal={arXiv preprint arXiv:2408.09675},
  year={2024}
}

Contact

If you have any questions or good supplyments for the advanced research, talk, or any form of relevant materials, please contact us and we appreciate for your contribute.

• Email [email protected](primary), and cc to [email protected], [email protected], [email protected]
• Github issues We would appreciate it if you reported any factual, technical and copyright issue you found in this repository.