From e9730cb48c13125f980dd9787e647d970a2b9574 Mon Sep 17 00:00:00 2001 From: samuelkuehnel <51356601+samuelkuehnel@users.noreply.github.com> Date: Mon, 13 Nov 2023 09:22:09 +0000 Subject: [PATCH 1/6] Research Pylot Planning/PAF 21-2 Perception --- .../research_PAF_23/Sprint0:_Research.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 doc/03_research/research_PAF_23/Sprint0:_Research.md diff --git a/doc/03_research/research_PAF_23/Sprint0:_Research.md b/doc/03_research/research_PAF_23/Sprint0:_Research.md new file mode 100644 index 00000000..fa7f0738 --- /dev/null +++ b/doc/03_research/research_PAF_23/Sprint0:_Research.md @@ -0,0 +1,60 @@ +# Sprint 0: Research Samuel Kühnel + +# Pylot + + +## Planning + +- 4 different options +- **Waypoint Planner**: Auto follows predefined waypoints. It recognizes traffic lights and stops at obstacles, but cannot avoid them +- **Freenet-Optimal-Trajecotry-Planner**: CPP code with Python wrapper ([GitHub](https://github.com/erdos-project/frenet_optimal_trajectory_planner)) +→ Predefined line that is used for orientation → Can avoid obstacles! +- **RRT\*-Planner**: RRT* algorithm to plan the path ([GitHub](https://github.com/erdos-project/rrt_star_planner)) + - Creates random nodes + - Adds nodes to the graph that are not blocked by objects on the road + - Generally terminates as soon as a node is found in the target area + - RRT*: Searches for the shortest path +- **Hybrid A\* planner**: Hybrid A* algorithm for route planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) + - Calculates the shortest path between two nodes from a graph + - Similar to Dijkstra's algorithm + - Nodes are estimated based on their costs and promising nodes are selected first + - Hybrid A* algorithm: Not always optimal solution, but in the neighborhood of the optimal solution. + +# PAF 21-2 + +# Perception + +## Obstacle detection + +- Detect objects via semantic lidar sensor +- Provides x and y coordinates, as well as distance value +- Additional information on position change in a time interval → Calculation of speed possible +- Detects the object and returns either the value "Vehicle" or "Pedestrian" + +## TrafficLightDetection + +![diagramm.png](https://github.com/ll7/paf21-2/raw/main/docs/imgs/trafficlightdetection_diagram.jpg) + +- **FusionCamera** saves images from **RGBCamera** and **DepthCamera** with timestamp and then synchronizes with **SegmentationCamera** +- Neural network based on [ResNet18](https://pytorch.org/hub/pytorch_vision_resnet/) (predefined PyTorch network) +- Generally only traffic lights up to 100m distance +- Canny algorithm to filter contours + +### Problems and solutions + +- Red background distorts traffic light phase detection → **Solution**: Narrow section of the traffic light image for phase detection +- Yellow painted traffic lights distort traffic light phase detection → **Solution**: Filter out red and green sections beforehand using masks and convert remaining image to grayscale and add masks again. +- **Problem without solution**: European traffic lights can sometimes not be recognized at the stop line. + +# Resumee + +### Perception + +- Status quo: LIDAR Sensor and Big Neural Network +- Possible focus: Using smaller (pretrained) models to improve overall performance fast +- Taking known issues into account + +### Planning + +- Currently decision tree for evaluating the current position +- Trying out different heuristics → already given as repo \ No newline at end of file From 9b19bcddb18016a71edba75f46109321a3c53006 Mon Sep 17 00:00:00 2001 From: samuelkuehnel <51356601+samuelkuehnel@users.noreply.github.com> Date: Mon, 13 Nov 2023 10:53:59 +0100 Subject: [PATCH 2/6] Update Sprint0:_Research.md Formatting --- .../research_PAF_23/Sprint0:_Research.md | 31 +++++++++---------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/doc/03_research/research_PAF_23/Sprint0:_Research.md b/doc/03_research/research_PAF_23/Sprint0:_Research.md index fa7f0738..9fa8c1ad 100644 --- a/doc/03_research/research_PAF_23/Sprint0:_Research.md +++ b/doc/03_research/research_PAF_23/Sprint0:_Research.md @@ -1,7 +1,6 @@ # Sprint 0: Research Samuel Kühnel -# Pylot - +## Pylot ## Planning @@ -10,28 +9,28 @@ - **Freenet-Optimal-Trajecotry-Planner**: CPP code with Python wrapper ([GitHub](https://github.com/erdos-project/frenet_optimal_trajectory_planner)) → Predefined line that is used for orientation → Can avoid obstacles! - **RRT\*-Planner**: RRT* algorithm to plan the path ([GitHub](https://github.com/erdos-project/rrt_star_planner)) - - Creates random nodes - - Adds nodes to the graph that are not blocked by objects on the road - - Generally terminates as soon as a node is found in the target area - - RRT*: Searches for the shortest path + - Creates random nodes + - Adds nodes to the graph that are not blocked by objects on the road + - Generally terminates as soon as a node is found in the target area + - RRT*: Searches for the shortest path - **Hybrid A\* planner**: Hybrid A* algorithm for route planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) - - Calculates the shortest path between two nodes from a graph - - Similar to Dijkstra's algorithm - - Nodes are estimated based on their costs and promising nodes are selected first - - Hybrid A* algorithm: Not always optimal solution, but in the neighborhood of the optimal solution. + - Calculates the shortest path between two nodes from a graph + - Similar to Dijkstra's algorithm + - Nodes are estimated based on their costs and promising nodes are selected first + - Hybrid A* algorithm: Not always optimal solution, but in the neighborhood of the optimal solution. -# PAF 21-2 +## PAF 21-2 -# Perception +### Perception -## Obstacle detection +### Obstacle detection - Detect objects via semantic lidar sensor - Provides x and y coordinates, as well as distance value - Additional information on position change in a time interval → Calculation of speed possible - Detects the object and returns either the value "Vehicle" or "Pedestrian" -## TrafficLightDetection +### TrafficLightDetection ![diagramm.png](https://github.com/ll7/paf21-2/raw/main/docs/imgs/trafficlightdetection_diagram.jpg) @@ -46,7 +45,7 @@ - Yellow painted traffic lights distort traffic light phase detection → **Solution**: Filter out red and green sections beforehand using masks and convert remaining image to grayscale and add masks again. - **Problem without solution**: European traffic lights can sometimes not be recognized at the stop line. -# Resumee +## Resumee ### Perception @@ -57,4 +56,4 @@ ### Planning - Currently decision tree for evaluating the current position -- Trying out different heuristics → already given as repo \ No newline at end of file +- Trying out different heuristics → already given as repo From 9f25dceee12e043fe85dfaa8cef3a6f30932feca Mon Sep 17 00:00:00 2001 From: samuelkuehnel <51356601+samuelkuehnel@users.noreply.github.com> Date: Mon, 13 Nov 2023 11:26:54 +0100 Subject: [PATCH 3/6] File renamed --- ...esearch_Pylot-Planning_PAF21-Perception.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md diff --git a/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md b/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md new file mode 100644 index 00000000..9fa8c1ad --- /dev/null +++ b/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md @@ -0,0 +1,59 @@ +# Sprint 0: Research Samuel Kühnel + +## Pylot + +## Planning + +- 4 different options +- **Waypoint Planner**: Auto follows predefined waypoints. It recognizes traffic lights and stops at obstacles, but cannot avoid them +- **Freenet-Optimal-Trajecotry-Planner**: CPP code with Python wrapper ([GitHub](https://github.com/erdos-project/frenet_optimal_trajectory_planner)) +→ Predefined line that is used for orientation → Can avoid obstacles! +- **RRT\*-Planner**: RRT* algorithm to plan the path ([GitHub](https://github.com/erdos-project/rrt_star_planner)) + - Creates random nodes + - Adds nodes to the graph that are not blocked by objects on the road + - Generally terminates as soon as a node is found in the target area + - RRT*: Searches for the shortest path +- **Hybrid A\* planner**: Hybrid A* algorithm for route planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) + - Calculates the shortest path between two nodes from a graph + - Similar to Dijkstra's algorithm + - Nodes are estimated based on their costs and promising nodes are selected first + - Hybrid A* algorithm: Not always optimal solution, but in the neighborhood of the optimal solution. + +## PAF 21-2 + +### Perception + +### Obstacle detection + +- Detect objects via semantic lidar sensor +- Provides x and y coordinates, as well as distance value +- Additional information on position change in a time interval → Calculation of speed possible +- Detects the object and returns either the value "Vehicle" or "Pedestrian" + +### TrafficLightDetection + +![diagramm.png](https://github.com/ll7/paf21-2/raw/main/docs/imgs/trafficlightdetection_diagram.jpg) + +- **FusionCamera** saves images from **RGBCamera** and **DepthCamera** with timestamp and then synchronizes with **SegmentationCamera** +- Neural network based on [ResNet18](https://pytorch.org/hub/pytorch_vision_resnet/) (predefined PyTorch network) +- Generally only traffic lights up to 100m distance +- Canny algorithm to filter contours + +### Problems and solutions + +- Red background distorts traffic light phase detection → **Solution**: Narrow section of the traffic light image for phase detection +- Yellow painted traffic lights distort traffic light phase detection → **Solution**: Filter out red and green sections beforehand using masks and convert remaining image to grayscale and add masks again. +- **Problem without solution**: European traffic lights can sometimes not be recognized at the stop line. + +## Resumee + +### Perception + +- Status quo: LIDAR Sensor and Big Neural Network +- Possible focus: Using smaller (pretrained) models to improve overall performance fast +- Taking known issues into account + +### Planning + +- Currently decision tree for evaluating the current position +- Trying out different heuristics → already given as repo From c5e8490625c3ed656679f1dd56e652704af7c4cb Mon Sep 17 00:00:00 2001 From: samuelkuehnel <51356601+samuelkuehnel@users.noreply.github.com> Date: Mon, 13 Nov 2023 11:27:34 +0100 Subject: [PATCH 4/6] Old file removed --- .../research_PAF_23/Sprint0:_Research.md | 59 ------------------- 1 file changed, 59 deletions(-) delete mode 100644 doc/03_research/research_PAF_23/Sprint0:_Research.md diff --git a/doc/03_research/research_PAF_23/Sprint0:_Research.md b/doc/03_research/research_PAF_23/Sprint0:_Research.md deleted file mode 100644 index 9fa8c1ad..00000000 --- a/doc/03_research/research_PAF_23/Sprint0:_Research.md +++ /dev/null @@ -1,59 +0,0 @@ -# Sprint 0: Research Samuel Kühnel - -## Pylot - -## Planning - -- 4 different options -- **Waypoint Planner**: Auto follows predefined waypoints. It recognizes traffic lights and stops at obstacles, but cannot avoid them -- **Freenet-Optimal-Trajecotry-Planner**: CPP code with Python wrapper ([GitHub](https://github.com/erdos-project/frenet_optimal_trajectory_planner)) -→ Predefined line that is used for orientation → Can avoid obstacles! -- **RRT\*-Planner**: RRT* algorithm to plan the path ([GitHub](https://github.com/erdos-project/rrt_star_planner)) - - Creates random nodes - - Adds nodes to the graph that are not blocked by objects on the road - - Generally terminates as soon as a node is found in the target area - - RRT*: Searches for the shortest path -- **Hybrid A\* planner**: Hybrid A* algorithm for route planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) - - Calculates the shortest path between two nodes from a graph - - Similar to Dijkstra's algorithm - - Nodes are estimated based on their costs and promising nodes are selected first - - Hybrid A* algorithm: Not always optimal solution, but in the neighborhood of the optimal solution. - -## PAF 21-2 - -### Perception - -### Obstacle detection - -- Detect objects via semantic lidar sensor -- Provides x and y coordinates, as well as distance value -- Additional information on position change in a time interval → Calculation of speed possible -- Detects the object and returns either the value "Vehicle" or "Pedestrian" - -### TrafficLightDetection - -![diagramm.png](https://github.com/ll7/paf21-2/raw/main/docs/imgs/trafficlightdetection_diagram.jpg) - -- **FusionCamera** saves images from **RGBCamera** and **DepthCamera** with timestamp and then synchronizes with **SegmentationCamera** -- Neural network based on [ResNet18](https://pytorch.org/hub/pytorch_vision_resnet/) (predefined PyTorch network) -- Generally only traffic lights up to 100m distance -- Canny algorithm to filter contours - -### Problems and solutions - -- Red background distorts traffic light phase detection → **Solution**: Narrow section of the traffic light image for phase detection -- Yellow painted traffic lights distort traffic light phase detection → **Solution**: Filter out red and green sections beforehand using masks and convert remaining image to grayscale and add masks again. -- **Problem without solution**: European traffic lights can sometimes not be recognized at the stop line. - -## Resumee - -### Perception - -- Status quo: LIDAR Sensor and Big Neural Network -- Possible focus: Using smaller (pretrained) models to improve overall performance fast -- Taking known issues into account - -### Planning - -- Currently decision tree for evaluating the current position -- Trying out different heuristics → already given as repo From 2b974aa0a0b2d17f1e6be8ff476b57097251ab1b Mon Sep 17 00:00:00 2001 From: samuelkuehnel <51356601+samuelkuehnel@users.noreply.github.com> Date: Mon, 13 Nov 2023 13:09:36 +0100 Subject: [PATCH 5/6] Update Research_Pylot-Planning_PAF21-Perception.md: GIFs added --- .../Research_Pylot-Planning_PAF21-Perception.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md b/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md index 9fa8c1ad..b2e3d17c 100644 --- a/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md +++ b/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md @@ -8,17 +8,25 @@ - **Waypoint Planner**: Auto follows predefined waypoints. It recognizes traffic lights and stops at obstacles, but cannot avoid them - **Freenet-Optimal-Trajecotry-Planner**: CPP code with Python wrapper ([GitHub](https://github.com/erdos-project/frenet_optimal_trajectory_planner)) → Predefined line that is used for orientation → Can avoid obstacles! + +![freenet_gif](https://github.com/erdos-project/frenet_optimal_trajectory_planner/raw/master/img/fot2.gif) + - **RRT\*-Planner**: RRT* algorithm to plan the path ([GitHub](https://github.com/erdos-project/rrt_star_planner)) - Creates random nodes - Adds nodes to the graph that are not blocked by objects on the road - Generally terminates as soon as a node is found in the target area - RRT*: Searches for the shortest path + +![rrt_star_gif](https://github.com/erdos-project/rrt_star_planner/raw/master/img/rrtstar.gif) + - **Hybrid A\* planner**: Hybrid A* algorithm for route planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) - Calculates the shortest path between two nodes from a graph - Similar to Dijkstra's algorithm - Nodes are estimated based on their costs and promising nodes are selected first - Hybrid A* algorithm: Not always optimal solution, but in the neighborhood of the optimal solution. +![hybrid_astar_gif](https://github.com/erdos-project/hybrid_astar_planner/raw/master/img/straight_obstacle.gif) + ## PAF 21-2 ### Perception From 6c9ff734e6e4c6a231a6c8cb6a665276b4cf340a Mon Sep 17 00:00:00 2001 From: samuelkuehnel <51356601+samuelkuehnel@users.noreply.github.com> Date: Mon, 13 Nov 2023 13:11:47 +0100 Subject: [PATCH 6/6] Update Research_Pylot-Planning_PAF21-Perception.md: Typo removed --- .../Research_Pylot-Planning_PAF21-Perception.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md b/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md index b2e3d17c..33df4320 100644 --- a/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md +++ b/doc/03_research/research_PAF_23/Research_Pylot-Planning_PAF21-Perception.md @@ -11,7 +11,7 @@ ![freenet_gif](https://github.com/erdos-project/frenet_optimal_trajectory_planner/raw/master/img/fot2.gif) -- **RRT\*-Planner**: RRT* algorithm to plan the path ([GitHub](https://github.com/erdos-project/rrt_star_planner)) +- **RRT\*-Planner**: RRT* algorithm for path planning ([GitHub](https://github.com/erdos-project/rrt_star_planner)) - Creates random nodes - Adds nodes to the graph that are not blocked by objects on the road - Generally terminates as soon as a node is found in the target area @@ -19,7 +19,7 @@ ![rrt_star_gif](https://github.com/erdos-project/rrt_star_planner/raw/master/img/rrtstar.gif) -- **Hybrid A\* planner**: Hybrid A* algorithm for route planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) +- **Hybrid A\* planner**: Hybrid A* algorithm for path planning ([GitHub](https://github.com/erdos-project/hybrid_astar_planner)) - Calculates the shortest path between two nodes from a graph - Similar to Dijkstra's algorithm - Nodes are estimated based on their costs and promising nodes are selected first