From 3a5158cc2a8e58f81b9e1b6695a5e5d2d69c623f Mon Sep 17 00:00:00 2001 From: Sebastian Seitz Date: Thu, 31 Oct 2024 19:05:54 +0100 Subject: [PATCH 1/3] documented driving score metric --- doc/research/paf24/general/driving_score.md | 82 +++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 doc/research/paf24/general/driving_score.md diff --git a/doc/research/paf24/general/driving_score.md b/doc/research/paf24/general/driving_score.md new file mode 100644 index 00000000..d65c72c1 --- /dev/null +++ b/doc/research/paf24/general/driving_score.md @@ -0,0 +1,82 @@ +# Driving Score Computation + +**Summary:** The Driving score is the main performance metric of the agent and therefore has to be examined. + +- [A Starting Point for Metrics and Measurements](#a-starting-point-for-metrics-and-measurements) +- [Driving score](#driving-score) +- [Route completion](#route-completion) +- [Infraction penalty](#infraction-penalty) +- [Infractions](#infractions) +- [Off-road driving](#off-road-driving) +- [Additional Events](#additional-events) +- [Sources:](#sources) + +## A Starting Point for Metrics and Measurements + +The CARLA Leaderboard sets a public example for evaluation and comparison. +The driving proficiency of an agent can be characterized by multiple metrics. For this leaderboard, the CARLA team selected a set of metrics that help understand different aspects of driving. + +## Driving score + +\[\frac{1}{N}\sum^i_N R_i P_i\] + +- The main metric of the leaderboard, serving as an aggregate of the average route completion and the number of traffic infractions. Here $N$ stands for the number of routes, $R_i$ is the percentage of completion of the $i$-th route, and $P_i$ is the infraction penalty of the $i$-th route. + +## Route completion + +\[\frac{1}{N}\sum^i_N R_i\] + +- Percentage of route distance completed by an agent, averaged across $N$ routes. + +## Infraction penalty + +\[\prod_j^{ped, veh, ... stop} (p_j^i)^{n_{infractions}}\] + +- Aggregates the number of infractions triggered by an agent as a geometric series. Agents start with an ideal 1.0 base score, which is reduced by a penalty coefficient for every instance of these. + +## Infractions + +The CARLA leaderboard offers individual metrics for a series of infractions. Each of these has a penalty coefficient that will be applied every time it happens. Ordered by severity, the infractions are the following. + +- Collisions with pedestrians $0.50$ +- Collisions with other vehicles $0.60$ +- Collisions with static elements $0.65$ +- Running a red light $0.70$ +- Running a stop sign $0.80$ + +Some scenarios feature behaviors that can block the ego-vehicle indefinitely. These scenarios will have a timeout of 4 minutes after which the ego-vehicle will be released to continue the route. However, a penalty is applied when the time limit is breached + +- Scenario Timeout $0.70$ + +The agent is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will be compared with the speed of nearby vehicles. Failure to maintain a suitable speed will result in a penalty. +The penalty applied is dependent on the magnitude of the speed difference, up to the following value: + +- Failure to maintain speed $0.70$ + +The agent should yield to emergency vehicles coming from behind. Failure to allow the emergency vehicle to pass will incur a penalty: + +- Failure to yield to emergency vehicle $0.70$ + +Besides these, there is one additional infraction which has no coefficient, and instead affects the computation of route completion $(R_i)$. + +## Off-road driving + +If an agent drives off-road, that percentage of the route will not be considered towards the computation of the route completion score. + +## Additional Events + +Some events will interrupt the simulation, preventing the agent to continue. + +- Route deviation +If an agent deviates more than $30$ meters from the assigned route. +- Agent blocked +If an agent is blocked in traffic without taking any actions for $180$ simulation seconds. +- Simulation timeout +If no client-server communication can be established in $60$ seconds. +- Route timeout +This timeout is triggered if the simulation of a route takes more than the allowed time. This allowed time is computed by multiplying the route distance in meters by a factor of $0.8$. + +## Sources: + +- [Alpha Drive Carla Leaderboard Case Study](https://alphadrive.ai/industries/automotive/carla-leaderboard-case-study/) +- [Carla Leaderboard](https://leaderboard.carla.org/) From fcb6783bd503c9b1c2645d0ec3b3b2e57d60027e Mon Sep 17 00:00:00 2001 From: Sebastian Seitz Date: Thu, 31 Oct 2024 19:09:03 +0100 Subject: [PATCH 2/3] make sure linting passes --- doc/research/paf24/general/driving_score.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/research/paf24/general/driving_score.md b/doc/research/paf24/general/driving_score.md index d65c72c1..a9b68181 100644 --- a/doc/research/paf24/general/driving_score.md +++ b/doc/research/paf24/general/driving_score.md @@ -9,7 +9,7 @@ - [Infractions](#infractions) - [Off-road driving](#off-road-driving) - [Additional Events](#additional-events) -- [Sources:](#sources) +- [Sources](#sources) ## A Starting Point for Metrics and Measurements @@ -76,7 +76,7 @@ If no client-server communication can be established in $60$ seconds. - Route timeout This timeout is triggered if the simulation of a route takes more than the allowed time. This allowed time is computed by multiplying the route distance in meters by a factor of $0.8$. -## Sources: +## Sources - [Alpha Drive Carla Leaderboard Case Study](https://alphadrive.ai/industries/automotive/carla-leaderboard-case-study/) - [Carla Leaderboard](https://leaderboard.carla.org/) From d13de4e585ec0826eb12b63446f86ab7fa3d3e42 Mon Sep 17 00:00:00 2001 From: Sebastian Seitz Date: Fri, 1 Nov 2024 21:24:53 +0100 Subject: [PATCH 3/3] fixed formula formatting --- doc/research/paf24/general/driving_score.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/research/paf24/general/driving_score.md b/doc/research/paf24/general/driving_score.md index a9b68181..4eaeea7e 100644 --- a/doc/research/paf24/general/driving_score.md +++ b/doc/research/paf24/general/driving_score.md @@ -18,19 +18,19 @@ The driving proficiency of an agent can be characterized by multiple metrics. Fo ## Driving score -\[\frac{1}{N}\sum^i_N R_i P_i\] +$\frac{1}{N}\sum^i_N R_i P_i$ - The main metric of the leaderboard, serving as an aggregate of the average route completion and the number of traffic infractions. Here $N$ stands for the number of routes, $R_i$ is the percentage of completion of the $i$-th route, and $P_i$ is the infraction penalty of the $i$-th route. ## Route completion -\[\frac{1}{N}\sum^i_N R_i\] +$\frac{1}{N}\sum^i_N R_i$ - Percentage of route distance completed by an agent, averaged across $N$ routes. ## Infraction penalty -\[\prod_j^{ped, veh, ... stop} (p_j^i)^{n_{infractions}}\] +$\prod_j^{ped, veh, ... stop} (p_j^i)^{n_{infractions}}$ - Aggregates the number of infractions triggered by an agent as a geometric series. Agents start with an ideal 1.0 base score, which is reduced by a penalty coefficient for every instance of these.