Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

69 feature documentation of changes from leaderboard 20 #70

Merged

Conversation

samuelkuehnel
Copy link
Collaborator

Description

Documentation for Leaderboard-2.0 added

Fixes #69

Type of change

Does this PR introduce a breaking change?

no

Most important changes

Added markdown doc and provided CARLA python script to collect test/training data

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works (might be obsolete with CI later on)
  • New and existing unit tests pass locally with my changes (might be obsolete with CI later on)

@samuelkuehnel samuelkuehnel added the documentation Improvements or additions to documentation label Nov 18, 2023
@samuelkuehnel samuelkuehnel requested a review from MaxJa4 November 18, 2023 14:48
@samuelkuehnel samuelkuehnel linked an issue Nov 18, 2023 that may be closed by this pull request
1 task
Copy link

Simulation results

Metric Value
Avg. driving score 0.014142
Avg. route completion 0.205
Avg. infraction penalty 0.257169
Collisions with pedestrians 0.257169
Collisions with vehicles 0.0
Collisions with layout 36.155
Red lights infractions 72.309
Stop sign infractions 0.0
Off-road infractions 0.0
Route deviations 0.023
Route timeouts 0.0
Agent blocked 36.155
Yield emergency vehicles infractions 36.155
Scenario timeouts 0.0
Min speed infractions 36.155
undefined 0.0

@MaxJa4
Copy link
Collaborator

MaxJa4 commented Nov 18, 2023

Really like the new script, this can certainly help us develop faster. If I understand correctly, this enables to develop independently (planning can just use the perfect-run data from the script, so can acting too). Also gives perception the opportunity to benchmark its models without needing a already-done planning/acting stage 😁


### Szenarios and training database

The new leaderboard comes with a large set of (new) szenarios that the autonomous agent has to accomplish. Some examples are merging lanes, emergency vehilces or Accidents. For each of the new scenrios CARLA provides logs from a $100 \%$ finish.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • "vehicles"
  • "accidents"
  • "scenarios" (2x)
  • The percent sign is not rendering

Just some really small stuff :)

Copy link

Simulation results

Metric Value
Avg. driving score 0.014392
Avg. route completion 0.335
Avg. infraction penalty 0.245858
Collisions with pedestrians 0.245858
Collisions with vehicles 0.0
Collisions with layout 22.166
Red lights infractions 110.832
Stop sign infractions 0.0
Off-road infractions 0.0
Route deviations 0.029
Route timeouts 0.0
Agent blocked 22.166
Yield emergency vehicles infractions 22.166
Scenario timeouts 0.0
Min speed infractions 22.166
undefined 0.0

@samuelkuehnel samuelkuehnel merged commit 4a3dc32 into main Nov 19, 2023
3 checks passed
@MaxJa4 MaxJa4 deleted the 69-feature-documentation-of-changes-from-leaderboard-20 branch November 22, 2023 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[Feature]: Documentation of changes from Leaderboard 2.0
2 participants