366 examine how the driving score is computed #405
Conversation
WalkthroughThe changes introduce a comprehensive overview of the "Driving Score" computation in the document Changes
Assessment against linked issues
Possibly related PRs
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (5)
doc/research/paf24/general/driving_score.md (5)
1-4: Enhance the document summary.Consider expanding the summary to include:
- The purpose of this document (e.g., reference guide, technical specification)
- The target audience (e.g., developers, researchers)
- The relationship to the CARLA simulation environment
Example enhancement:
# Driving Score Computation -**Summary:** The Driving score is the main performance metric of the agent and therefore has to be examined. +**Summary:** This technical reference document explains how the Driving Score, the main performance metric for autonomous agents in the CARLA simulation environment, is computed. It serves as a comprehensive guide for developers and researchers to understand how agent performance is evaluated through route completion and various infractions.
21-23: Improve formula presentation and explanation.The driving score formula could be better formatted and explained:
- Use display math mode for better readability
- Add a more detailed explanation of the variables
Suggested enhancement:
-\[\frac{1}{N}\sum^i_N R_i P_i\] +\[ +\text{DrivingScore} = \frac{1}{N}\sum_{i=1}^N R_i P_i +\] +where: +- $N$ is the total number of routes evaluated +- $R_i$ is the completion percentage of route $i$ (between 0 and 1) +- $P_i$ is the cumulative infraction penalty for route $i$ (between 0 and 1)
41-59: Standardize infraction penalties format and add measurement criteria.Consider enhancing this section with:
- Consistent formatting for penalty values
- Specific criteria for each infraction
- Tabular format for better readability
Example enhancement:
| Infraction Type | Penalty Coefficient | Measurement Criteria | |-----------------|:------------------:|---------------------| | Collision with pedestrians | 0.50 | Any contact between agent and pedestrian | | Collision with vehicles | 0.60 | Any contact between agent and other vehicles | | Collision with static elements | 0.65 | Any contact with buildings, poles, etc. | | Running red light | 0.70 | Crossing stop line while signal is red | | Running stop sign | 0.80 | Failing to come to complete stop | | Scenario timeout | 0.70 | Exceeding 4-minute time limit | | Speed maintenance failure | 0.70 | Speed deviation from nearby traffic | | Emergency vehicle yield failure | 0.70 | Not yielding within X seconds |🧰 Tools
🪛 LanguageTool
[style] ~51-~51: ‘in keeping with’ might be wordy. Consider a shorter alternative.
Context: ...is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will ...(EN_WORDINESS_PREMIUM_IN_KEEPING_WITH)
68-78: Enhance timeout explanations with examples.Consider adding:
- Examples for each timeout scenario
- Explanation of the 0.8 factor in route timeout calculation
- Practical examples of route timeout calculations
Example addition:
For example, for a 1000-meter route: - Maximum allowed time = 1000 * 0.8 = 800 seconds - This allows for an average speed of 4.5 km/h (1.25 m/s)
79-82: Expand sources section with version information.Consider enhancing the sources section with:
- Access dates for external links
- CARLA version number this documentation applies to
- Additional relevant documentation or research papers
Example format:
## Sources - [Alpha Drive Carla Leaderboard Case Study](https://alphadrive.ai/industries/automotive/carla-leaderboard-case-study/) (accessed October 2024) - [Carla Leaderboard](https://leaderboard.carla.org/) (CARLA version X.Y.Z) - CARLA Documentation v0.9.13: Metrics and Evaluation
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
doc/research/paf24/general/driving_score.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
doc/research/paf24/general/driving_score.md
[style] ~51-~51: ‘in keeping with’ might be wordy. Consider a shorter alternative.
Context: ...is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will ...
(EN_WORDINESS_PREMIUM_IN_KEEPING_WITH)
🔇 Additional comments (1)
doc/research/paf24/general/driving_score.md (1)
62-65: Clarify off-road driving criteria.
The current description lacks specific details about:
- How "off-road" is defined (e.g., percentage of vehicle outside lane markings)
- How this percentage affects the route completion score
- Whether there's a tolerance threshold before penalties apply
Could you provide more specific details about these criteria?
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (4)
doc/research/paf24/general/driving_score.md (4)
19-24: Enhance formula explanation for better readability.Consider expanding the explanation to define variables more explicitly:
- What is the expected range of values for
$R_i$ (e.g., [0,1] or [0,100])?- Is
$P_i$ always ≤ 1.0 since it's described as starting from an "ideal 1.0 base score"?Would you like me to propose an expanded explanation that includes these details?
37-61: Consider adding context for penalty coefficients and timeout values.To improve understanding:
- Add rationale for the chosen penalty coefficients (e.g., why is pedestrian collision 0.50 vs vehicle collision 0.60?)
- Explain why 4 minutes was chosen as the timeout threshold
- Consider adding examples of how multiple infractions compound using the geometric series formula
🧰 Tools
🪛 LanguageTool
[style] ~51-~51: ‘in keeping with’ might be wordy. Consider a shorter alternative.
Context: ...is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will ...(EN_WORDINESS_PREMIUM_IN_KEEPING_WITH)
51-53: Simplify wording for clarity.The current phrase "maintain a minimum speed in keeping with nearby traffic" could be simplified.
Consider this revision:
-The agent is expected to maintain a minimum speed in keeping with nearby traffic. +The agent is expected to maintain a speed consistent with nearby traffic.🧰 Tools
🪛 LanguageTool
[style] ~51-~51: ‘in keeping with’ might be wordy. Consider a shorter alternative.
Context: ...is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will ...(EN_WORDINESS_PREMIUM_IN_KEEPING_WITH)
79-82: Consider expanding the sources section.The current sources focus solely on CARLA-specific implementations. Consider adding:
- Academic references about autonomous driving metrics
- Industry standards or benchmarks for driving performance evaluation
- Links to related CARLA documentation about the simulation environment
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
doc/research/paf24/general/driving_score.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
doc/research/paf24/general/driving_score.md
[style] ~51-~51: ‘in keeping with’ might be wordy. Consider a shorter alternative.
Context: ...is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will ...
(EN_WORDINESS_PREMIUM_IN_KEEPING_WITH)
🔇 Additional comments (3)
doc/research/paf24/general/driving_score.md (3)
1-13: LGTM! Well-structured document organization.
The table of contents is clear, comprehensive, and properly linked to all sections.
76-78: Clarify route timeout calculation.
The factor 0.8 used in route timeout calculation needs more context:
- What's the reasoning behind using 0.8 specifically?
- Does this mean the agent is expected to maintain an average speed of 1.25 m/s (1/0.8)?
- Consider adding an example calculation for better understanding.
1-82: Overall approval: Documentation meets objectives.
The document successfully fulfills its purpose of explaining the driving score computation, providing a comprehensive overview of all components and metrics. The mathematical formulas, penalty systems, and various scenarios are well-documented, making it a valuable reference for understanding the CARLA evaluation system.
🧰 Tools
🪛 LanguageTool
[style] ~51-~51: ‘in keeping with’ might be wordy. Consider a shorter alternative.
Context: ...is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will ...
(EN_WORDINESS_PREMIUM_IN_KEEPING_WITH)
Description
This change adds a file explaining the computation of the driving score.
Fixes #366
Type of change
Does this PR introduce a breaking change?
No.
Most important changes
changes file /doc/research/paf24/general/driving_score.md
Checklist:
Summary by CodeRabbit
New Features
Documentation