From 06c8bc02c92236454f55a2c50e69eea0e1732d7b Mon Sep 17 00:00:00 2001 From: Sebastian Seitz Date: Wed, 30 Oct 2024 17:15:36 +0100 Subject: [PATCH] get current state and document it #363 --- doc/research/paf24/general/current_state.md | 183 ++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100755 doc/research/paf24/general/current_state.md diff --git a/doc/research/paf24/general/current_state.md b/doc/research/paf24/general/current_state.md new file mode 100755 index 00000000..bffc2432 --- /dev/null +++ b/doc/research/paf24/general/current_state.md @@ -0,0 +1,183 @@ +# Current state of the simulation + +**Summary:** The current state of the simulation is assessed by doing three runs of 20 mins (real world time), where all mistakes or anomalies are written down. + +- [Goal](#goal) +- [Methodology](#methodology) +- [Observed Errors Grouped by Domains](#observed-errors-grouped-by-domains) + - [Infrastructure](#infrastructure) + - [Testing and Validation](#testing-and-validation) + - [Perception](#perception) + - [Localization and Mapping](#localization-and-mapping) + - [Decision-Making](#decision-making) + - [Path Planning](#path-planning) + - [Control](#control) +- [Raw notes](#raw-notes) + - [Run 1](#run-1) + - [Run 2](#run-2) + - [Run 3](#run-3) + +## Goal + +In order to understand the current state of the agent, it is crucial to assess the status quo and note the challenges it is faced with. + +## Methodology + +This assessment was done by three leaderboard runs in the CARLA simulator with the handover-state for PAF24. While doing so, all mistakes made by the agent have been noted, as well as possible anomalies occurring during the inspection. +After the review, the mistakes have been grouped by the roles defined in the project in order to make it easier to address the challenges in the respective domains. **Note:** Some mistakes overlap and communication is key when tackling these issues. + +## Observed Errors Grouped by Domains + +### Infrastructure + +These issues relate to foundational aspects of the simulation environment and underlying software stability: + +- **Simulator Performance Degradation:** + - Simulation slows down over time (from .33 to .29 rate), potentially impacting reaction times and sensor data processing. +- **Vehicle Despawning:** + - Random despawning of cars and potential timeout for stuck vehicles may interfere with the agent’s perception and response. + +--- + +### Testing and Validation + +These errors highlight the gaps in the testing and validation process, particularly areas that may need further testing to ensure proper functioning in the real environment: + +- **Consistency in Object Detection:** + - Image segmentation flickering (e.g., police car with indicators), suggesting inadequate validation for dynamic objects with flashing lights. +- **Vision Node Stability:** + - Vision node appears to freeze occasionally, indicating possible untested scenarios or bugs in the perception pipeline. +- **Unrealistic Emergency Braking and Recovery Testing:** + - Unstable lane holding and recovery, resulting in inappropriate emergency braking maneuvers, suggests insufficient validation in complex recovery scenarios. +- **Misclassification of Tree Trunks:** + - Trees being detected as cars, indicating the need for validation of object detection in diverse environmental conditions. + +--- + +### Perception + +Errors within perception involve how the agent senses and understands its surroundings: + +- **Object Misclassification and Collision:** + - Tree trunks mistakenly detected as cars. + - Crashes into bikers and parked cars, suggesting perception failures in identifying and avoiding static and moving obstacles. +- **Segmentation and Detection Instability:** + - Vision node freezing. + - Flickering segmentation for objects like police cars with indicators. +- **Lane Detection and Holding Errors:** + - Difficulty in stable lane holding, leading to unexpected lane deviations and emergency braking. + - Misinterpretation of open car doors, causing lane intrusions without sufficient clearance. + +--- + +### Localization and Mapping + +Issues with localization and mapping involve understanding and positioning within the environment: + +- **Positioning Errors in Turns:** + - Turns are too wide, leading the agent onto the walkway, indicating potential localization issues in tight maneuvers. +- **Lane Holding and Position Drift:** + - Unstable lane holding with constant left and right drifting suggests potential mapping or localization inaccuracies. + +--- + +### Decision-Making + +Errors in decision-making relate to the agent's ability to make appropriate choices in response to various scenarios: + +- **Right of Way Violations:** + - Fails to yield to oncoming traffic when turning left and when merging into traffic. + - Ignores open car doors when passing parked cars, causing dangerous close passes. +- **Erroneous Stopping and Acceleration:** + - Stops unnecessarily at green lights and struggles to resume smoothly after stopping. + - Abrupt stopping and starting at green lights, potentially due to aggressive speed control. +- **Repeated Mistakes in Overtaking and Lane Changes:** + - Treats temporary parked cars as regular vehicles to overtake without checking oncoming traffic, leading to unsafe lane changes. + +--- + +### Path Planning + +Path planning issues include errors in determining the correct and safest path: + +- **Incorrect Overtaking Paths:** + - Attempts to overtake trees and temporary parked cars without considering oncoming traffic, showing flaws in path generation. +- **Wide Turning Paths:** + - Takes overly wide turns that lead to walkway intrusions. +- **Aggressive Lane Changes:** + - Lane change planning is overly aggressive, causing the vehicle to abruptly veer, triggering emergency stops to avoid collisions. + +--- + +### Control + +Control-related issues concern the vehicle’s execution of planned actions, like maintaining speed and stability: + +- **Abrupt and Aggressive Speed Control:** + - Speed controller is too aggressive when accelerating from green lights, leading to abrupt stopping and starting. +- **Instability in Lane Holding:** + - Inconsistent lane holding, particularly after getting unstuck, results in unexpected deviations onto walkways. +- **Inconsistent Recovery Behavior:** + - Repeatedly gets stuck in various situations (e.g., speed limit signs or temporary parked cars) and fails to recover smoothly, indicating control issues in re-engaging the driving path. + +--- + +## Raw notes + +Here are the raw notes in case misunderstandings have been made when grouping the mistakes + +### Run 1 + +- Scared to get out of parking spot +- lane not held causing problems when avoiding open car door +- stopping for no apparent reason +- does not keep lane (going left and right) +- driving into still standing car at red light +- impatient when waiting for light to turn green (after the crash, going back and forth) +- abrupt stopping and going when light turns green without reason → speed controller too aggressive? +- Problems to keep lane is causing emergency(?) brake maneuvers +- vision node seems to be frozen ? +- Detects bikers, crashes into them nonetheless +- lane change very aggressive causing emergency stop in order to not go into oncoming traffic +- gets stuck as a result +- simulator despawns cars randomly +- left turn does not give way to oncoming traffic when seeing them +- does the turn too wide, gets onto walkway +- simulation gets slower as time progresses, started at .33 rate, now at .29 +- gets stuck in front of speed limit sign after doing turn too wide +- gets unstuck, lane holding too aggressive goes onto walkway again (integrator windup while being stuck?) +- gets stuck again (→ unstuck behavior bad) +- when getting unstuck, merges onto street without giving way to traffic on the road +- drives into oncoming traffic, traffic on the same lane overtakes on the right side and does not stop +- really stuck now + +### Run 2 + +- merges without giving way to traffic +- does not respect open car door +- crashes into car in front when going after stop at red light +- stops at green light +- crashes into bikers +- kid runs onto street, agent crashes into oncoming traffic, gets stuck +- nudges away from the car it crashed into +- is now free but does not move +- crashes again +- police car with indicators on standing on the side is crashed into +- image segmentation for police car seems to be flickering +- tree trunk has bounding box (are trees detected as cars?) + +### Run 3 + +- does not give way when exiting a parking spot +- LIDAR detects floor +- trajectory for overtaking is wrong / no overtake needed +- stops without reason +- tries to "overtake" tree (detects tree as car) +- playback ration temperature dependent likely +- after emergency brake stops too long +- left turn doesn't give way to oncoming traffic +- recovery leads to oncoming traffic (left turn situation maybe doesn't recognize street?) 9 min +- temporary parked car with indicators on counts as normal overtake (does not check oncoming traffic) +- temporary parked car with indicators is the crux +- Despawn time of cars ? Cars despawn when stuck → over time limit ? +- Trajectory correctly generated, just too deep in the mistakes