A curated list of all awesome things related to self-driving car.
- Opensource code
- Courses
- Papers & Blogs
- Datasets & Benchmarks
- Algorithms
- Systems
- Cloud service
- Hardware
- Big Players
- Legislation
First, let's look at the technology stack for autonomous driving. In order to understand the full stack of autonomous driving. After that, you can learn the corresponding skill tree.
- apollo - Apollo is an open source autopilot platform that contains almost everything. Including hardware, systems, vehicle platforms, cloud services, etc. You can quickly build a self-driving system of your own by Apollo.
- autoware - The original Autoware project built on ROS 1. Launched as a research and development platform for autonomous driving technology.
- openpilot - Openpilot is an open source driver agent. Use the iphone to control the car, which provides adaptive cruise control (ACC) and lane keeping assist (LKAS).
- ROS - The Robot Operating System (ROS) is a flexible framework for writing robot software. It is a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behavior across a wide variety of robotic platforms.
- OpenCV library - OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.
- Point Cloud Library - Point Cloud Library (PCL) is a standalone, large open project for 2D / 3D imagery and point cloud processing. Widely used to process laser point cloud data.
- TensorFlow - TensorFlow is an open source software library for numerical computation using data flow graphs. Used for automatic driving perception and prediction.
- Udacity Self-Driving Car Nanodegree - Udacity's flagship program is sponsored by many self-driving car hiring partners. The nanodegree program includes 3 terms: 1) Term 1: Computer Vision and Deep Learning, 2) Term 2: Sensor Fusion, Localization, and Control, and 3) Term 3: Path Planning, Elective, and Systems. Each term costs $800.
- MIT 6.S094: Deep Learning for Self-Driving Cars - This class is an introduction to the practice of deep learning through the applied theme of building a self-driving car. It is open to beginners and is designed for those who are new to machine learning, but it can also benefit advanced researchers in the field looking for a practical overview of deep learning methods and their application. By the way, it's free!
- BitTiger Build Your Own Autonomous Vehicle Mastery Program - Two weeks of live classes in Bay Area taught by engineers from Vector.ai on building a self-driving mini car from ground up. Topics include deep learning, ROS, sensors, computer vision, localization, mapping and control. The program costs $7,000.
- Apollo Autopilot Introduction - Note: it's a Chinese tutorial. This is an open class between Baidu and Peking University. It is very comprehensive and detailed, which is a good introductory course.
[1] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, Kazuya Takeda. A Survey of Autonomous Driving: Common Practices and Emerging Technologies arXiv:1906.05113 arXiv:1906.05113v2 (2020). [pdf] (Very very comprehensive introduction) ⭐⭐⭐⭐⭐
[2] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, Gigel Macesanu. A Survey of Deep Learning Techniques for Autonomous Driving arXiv:1910.07738v2 (2020). [pdf] (Very very comprehensive introduction) ⭐⭐⭐⭐⭐
[3] Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro etc. Self-Driving Cars: A Survey arXiv:1901.04407v2 (2019). [pdf] (Very comprehensive introduction) ⭐⭐⭐⭐
[4] Jesse Levinson, Jake Askeland, Jan Becker etc. Towards Fully Autonomous Driving: Systems and Algorithms DOI: 10.1109/IVS.2011.5940562 (2011). [pdf] (Very very comprehensive introduction) ⭐⭐⭐⭐⭐
[1] ZENRIN Co.,Ltd Highly Efficient HD Map Creation: Accelerating Mapping Process with GPUs An introduction PPT (2017). [pdf] (Very very comprehensive introduction) ⭐⭐⭐⭐⭐
[1] Sampo Kuutti, Saber Fallah, Konstantinos Katsaros, Mehrdad Dianati, Francis Mccullough, Alexandros Mouzakitis A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications 10.1109/JIOT.2018.2812300 (2017). [pdf] (Very comprehensive survey) ⭐⭐⭐⭐⭐
[2] Guillaume Bresson, Zayed Alsayed, Li Yu, Sébastien Glaser Simultaneous localization and mapping: A survey of current trends in autonomous driving 10.1109/TIV.2017.2749181 (2017). [pdf] (Very comprehensive survey) ⭐⭐⭐⭐⭐
- Robust and Precise Vehicle Localization based on Multi-sensor Fusion in Diverse City Scenes - Present a robust and precise localization system that achieves centimeter-level localization accuracy in disparate city scenes by sensor fusion.
- Map-Based Precision Vehicle Localization in Urban Environments -
- Robust Vehicle Localization in Urban Environments Using Probabilistic Maps -
- Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art - State-of-the-art survey on computer vision-related problems datasets and methods for self-driving cars.
- Object Detection in 20 Years: A Survey - 2019
- Object Detection With Deep Learning: A Review -
- 50 Years of object recognition: Directions forward -
- Deep Learning for Generic Object Detection: A Survey -
[1] Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Xiaowei Zhao, Tae-Kyun Kim Multiple Object Tracking: A Literature Review arXiv:1409.7618v4 (2017). [pdf] (Very comprehensive survey) ⭐⭐⭐⭐⭐
- Multisensor data fusion: A review of the state-of-the-art
- A Review of Data Fusion Techniques
- A COMPREHENSIVE REVIEW OF THE MULTI-SENSOR DATA FUSION ARCHITECTURES
- A Survey of Multisensor Fusion Techniques, Architectures and Methodologies
[1] Florin Leon, Marius Gavrilescu. A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving arXiv:1909.07707v1 (2019). [pdf] (Very very comprehensive review) ⭐⭐⭐⭐⭐
-
A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles - A Survey.
-
A Review of Motion Planning Techniques for Automated Vehicles -
-
ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst - Waymo's paper about how to train a policy for autonomous driving via imitation learning that is robust enough to drive a real vehicle.
-
Baidu Apollo EM Motion Planner - A real-time motion planning system based on the Baidu Apollo (open source) autonomous driving platform.
- A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles - A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles.
- End to End Learning for Self-Driving Cars - 2016 NVIDIA
- An Introduction to LIDAR - Awesome introduction by Voyage about the key sensor of self-driving cars.
- Learning a Driving Simulator - comma.ai's approach for self-driving cars is based on an agent that learns to clone driver behaviors and plans maneuvers by simulating future events in the road. This paper investigates variational autoencoders with classical and learned cost functions using generative adversarial networks for embedding road frames. A transition model is learned in the embedded space using action conditioned Recurrent Neural Networks (RNNs).
- 16 Questions About Self-Driving Cars - a16z's Frank Chen goes over the 16 most commonly asked questions, technical and non-technical, about self-driving cars.
- Ways to think about cars - Awesome blog post by a16z's Benedict Evans on electric cars, on-demand car services, and self-driving cars.
- The Third Transportation Revolution - Awesome blog post by John Zimmer on Lyft's vision for self-driving cars. Spoiler alert, John predicts self-driving cars will account for the majority of Lyft rides within 5 years. And by 2025, private car ownership will all-but end in major U.S. cities.
- Cars and second order consequences - Benedict Evans on the impact of electric and autonomy on cars and beyond.
- KITTI Vision Benchmark Suite [Images] - Large vision benchmark dataset with objection detection evaluation training/testing images and leaderboard on cars and pedestrians.
- Cityscapes - Semantic, instance-wise, dense pixel annotations of 30 classes.
- comma.ai's Driving Dataset [Videos] - Seven and a quarter hours (~ 80 GB) of largely highway driving. With this dataset, comma.ai's founder George Hotz trained a self-driving car all by himself.
- Udacity's Driving Dataset [Videos] - Eight hours (over 280 GB) of driving data collected for their open source self-driving car challenges. Udacity also provides convenient scripts to port the data.
- Washington DC's Lidar Data - Lidar point cloud of the entire Washington DC area is made available by the District of Columbia’s Office of the Chief Technology Officer (OCTO).
- Apolloscape - Apolloscape provides images with 10x higher resolution and pixel-level annotation. And also Provides multiple levels of scene complexity.
- nuScenes - The nuScenes dataset (pronounced /nuːsiːnz/) is a public large-scale dataset for autonomous driving provided by nuTonomy-Aptiv.
- Waymo Open Dataset - The Waymo Open Dataset is comprised of high-resolution sensor data collected by Waymo self-driving cars in a wide variety of conditions.
- STSD - More than 20 000 images with 20% labeled, Contains 3488 traffic signs.
- LISA - 7855 annotations on 6610 frames.
- Tsinghua-Tencent 100K - 100000 images containing 30000 traffic-sign instances.
- German Traffic Sign [Images] - More than 50,000 images and 40 classes of traffic signs.
- Udacity's Self-Driving Car Simulator - This simulator is built for Udacity's Self-Driving Car Nanodegree to teach students how to train cars how to navigate road courses using deep learning. It is used for the project of Behavioral Cloning.
- Microsoft's AirSim - An open-source and cross platform simulator built for drones and other vehicles. AirSim is designed as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles.
- MIT's Moral Machine - Moral machine provides a "platform for 1) building a crowd-sourced picture of human opinion on how machines should make decisions when faced with moral dilemmas, and 2) crowd-sourcing assembly and discussion of potential scenarios of moral consequence". If you are a fan of the trolley problem, you can't miss this.
- MIT's Google Self-Driving Car Simulator - Self-driving car simulated completely by visual programming language Scratch.
- Carla - CARLA has been developed from the ground up to support development, training, and validation of autonomous driving systems.
- Lgsvl - The LGSVL Simulator is a simulator that facilitates testing and development of autonomous driving software systems. The LGSVL simulator is developed by the Advanced Platform Lab at the LG Electronics America R&D Center, formerly the LG Silicon Valley Lab.
If I have seen further it is by standing on ye sholders of Giants.
- Isaac Newton
| Waymo | Cruise Automation | Pony.ai | Baidu | Nuro |
| Zoox | Lyft | Autox | Mercedes Benz | Aurora |
| Apple | NVIDIA | AImotive | WeRide | Drive.ai |
| SF Motors/Seres | Nullmax | Nissan | SAIC | Qualcomm |
- Arizona - Executive order 2015-09 was signed to direct all state agencies to undertake any necessary steps to support the testing and operation of self-driving vehicles on public roads within Arizona. The executive order also specifies that in Arizona the operator of a self-driving vehicle does not have to be physically inside the vehicle. The vehicle can be directed remotely in self-driving mode.
- Colorado - Colorado passes first law to regulate self-driving cars. Companies who plan to test self-driving cars in Colorado need to first check in with the state Department of Transportation and State Patrol.
- California - Application required for testing self-driving cars. Manufacturers are required to provide accident reports and disengagement reports. The regulations for post-testing deployment of self-driving cars have been drafted and will establish the requirements for manufacturers to meet prior to operation on California’s on public roads.
- New York - It is announced on May 10, 2017 that the state is accepting application for self-driving car testing. The pilot testing program is included in FY 2018 and can be further extended.
- Texas - Bill SB 2205 is proposed to implement minimum safety requirements and accelerate testing of self-driving cars on public roads. The bill passed the Texas Senate Transportation Committee and it is currently pending Senate debate and referral to the House.
- Virginia - Arguably the most friendly state to self-driving cars with no application or permit required and $25M per year fund set to facilitate self-driving cars. Virginia Automated Corridors is announced to offer self-driving car developers the opportunity to test their technologies on Virginia roads covering more than 70 miles of interstates and arterials in the Northern Virginia region.
- Singapore -
- China -
- South Korea -
- Japan -
- Europe -