Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMCL core dumps due to a race condition when used with the map server. #4537

Closed
techtasie opened this issue Jul 15, 2024 · 9 comments · Fixed by #4605
Closed

AMCL core dumps due to a race condition when used with the map server. #4537

techtasie opened this issue Jul 15, 2024 · 9 comments · Fixed by #4605
Labels
bug Something isn't working

Comments

@techtasie
Copy link

Bug report

Required Info:

  • Operating System:
    • fedora 40
  • ROS2 Version: jazzy
    • jazzy
  • Version or commit hash: 1.3.1
    • from source: no changes from the source.
  • DDS implementation:
    • cyclonedds

Steps to reproduce issue

In the core dump, you see that map_ can be a null pointer and therefore you get an unhandled null pointer exception.
Uncommenting those lines circumvented the crash for us:

diff --color -ur navigation2-release-release-jazzy-nav2_amcl-1.3.1-1/src/amcl_node.cpp navigation2-release-release-jazzy-nav2_amcl-1.3.1-1-new/src/amcl_node.cpp
--- navigation2-release-release-jazzy-nav2_amcl-1.3.1-1/src/amcl_node.cpp	2024-06-25 20:01:09.000000000 +0200
+++ navigation2-release-release-jazzy-nav2_amcl-1.3.1-1-new/src/amcl_node.cpp	2024-07-15 17:43:12.792477289 +0200
@@ -540,13 +540,13 @@
       global_frame_id_.c_str());
     return;
   }
-  if (abs(msg->pose.pose.position.x) > map_->size_x ||
-    abs(msg->pose.pose.position.y) > map_->size_y)
-  {
-    RCLCPP_ERROR(
-      get_logger(), "Received initialpose from message is out of the size of map. Rejecting.");
-    return;
-  }
+  // if (abs(msg->pose.pose.position.x) > map_->size_x ||
+  //   abs(msg->pose.pose.position.y) > map_->size_y)
+  // {
+  //   RCLCPP_ERROR(
+  //     get_logger(), "Received initialpose from message is out of the size of map. Rejecting.");
+  //   return;
+  // }
 
   // Overriding last published pose to initial pose
   last_published_pose_ = *msg;

There is a race condition between the receiving of the map topic subscriber and setting the initial pose. On our robot, this race condition almost always happens but could be hard to reproduce on other setups.

Expected behavior

AMCL localization at initial pose or at least not crashing.

Actual behavior

AMCL coredumps

Additional information

I have attached our core dump. For the core dump, I rebuilt only AMCL in Debug mode; everything else was built as ReleaseWithDebug.
core.amcl.1000.c11f12581e0949ccb1dcd9e3d30b45d0.13740.1721053490000000.tar.gz
We are using a custom map server that may experience a slight delay when starting up. Therefore, AMCL could be launched without a map.

@SteveMacenski
Copy link
Member

That was recently added in #4416 by @GoesM during his fuzzing testing.

That makes sense, I think we should check if map_ != nullptr as part of that if statement before dereferencing it. @techtasie can you submit that change and make sure it solves your issue?

i.e.

if (map_ && (abs(msg->pose.pose.position.x) > map_->size_x || abs(msg->pose.pose.position.y) > map_->size_y))

@SteveMacenski SteveMacenski added the bug Something isn't working label Jul 15, 2024
@techtasie
Copy link
Author

@SteveMacenski Thank you for your fast reply I will test it thoroughly tomorrow during robocup.

@GoesM
Copy link
Contributor

GoesM commented Jul 16, 2024

Oh, so sorry for that! This is entirely caused by my mistake! I didn't notice that this map might be a null pointer.

@techtasie
Copy link
Author

Don't we need to still check for the nullptr like this?

  if ( (!first_map_received_) || (map_ && (abs(msg->pose.pose.position.x) > map_->size_x ||
    abs(msg->pose.pose.position.y) > map_->size_y))
  {
    RCLCPP_ERROR(
      get_logger(), "Received initialpose from message is out of the size of map. Rejecting.");
    return;
  }

@techtasie
Copy link
Author

I thing the way it is supposed to work is that the initial pose fires directly after the map has been received. Because in some circumstances you want to set the initial pose but get the map a second or so after amcl launch.

@GoesM
Copy link
Contributor

GoesM commented Jul 16, 2024

I thing the way it is supposed to work is that the initial pose fires directly after the map has been received. Because in some circumstances you want to set the initial pose but get the map a second or so after amcl launch.

I see. Thanks for your share √ 😀

@SteveMacenski
Copy link
Member

I think in that case we just need to check for map_ != nullptr, no? That would have been the behavior prior to the change (i.e. could set the initial pose before a map) so I think that's solid to continue to do

@GoesM
Copy link
Contributor

GoesM commented Jul 16, 2024

That would have been the behavior prior to the change

no problem, I misunderstood something before and I've understood it now.

@SteveMacenski
Copy link
Member

#4605 to resolve. thanks for the note @techtasie !

SteveMacenski added a commit that referenced this issue Aug 24, 2024
* Updated README Table once Jazzy jobs turn over (#4482)

* add new Jazzy matrix

* missing header

* test toolg

* retry

* done!

* trim

* trim

* fix OS[0]

* shutdown services in destructor of `ClearCostmapService` (#4495)

Signed-off-by: GoesM_server <[email protected]>
Co-authored-by: GoesM_server <[email protected]>

* adjusting backup speed to be more reasonable (#4501)

* Adding Costmap filters to system tests and cleaning up old gazebo classic files (#4502)

* removing old files

* removing old refs to gazebo classic

* porting test body

* including in root

* Dock panel (#4458)

* Initial docking panel

Signed-off-by: Alberto Tudela <[email protected]>

* Only one goal status

Signed-off-by: Alberto Tudela <[email protected]>

* Added dock  pose

Signed-off-by: Alberto Tudela <[email protected]>

* Fix size of text

Signed-off-by: Alberto Tudela <[email protected]>

* Update rviz

Signed-off-by: Alberto Tudela <[email protected]>

* Update rviz config

Signed-off-by: Alberto Tudela <[email protected]>

---------

Signed-off-by: Alberto Tudela <[email protected]>

* Fix default view (#4504)

* Fix logo in nav2 panel (#4505)

* Fix logo in nav2 panel

Signed-off-by: Alberto Tudela <[email protected]>

* Move icon

Signed-off-by: Alberto Tudela <[email protected]>

---------

Signed-off-by: Alberto Tudela <[email protected]>

* Fix world to map coordinate conversion (#4506)

Signed-off-by: HovorunBh <[email protected]>

* Update README.md

Signed-off-by: Steve Macenski <[email protected]>

* Add dock id (#4511)

* Implement dock id

Signed-off-by: redvinaa <[email protected]>

* Update tests

Signed-off-by: redvinaa <[email protected]>

* Update docs

Signed-off-by: redvinaa <[email protected]>

* Fix virtual override error

Signed-off-by: redvinaa <[email protected]>

---------

Signed-off-by: redvinaa <[email protected]>

* fix(nav2_costmap_2d): make obstacle layer not current on enabled toggle (#4507)

Signed-off-by: Kemal Bektas <[email protected]>
Co-authored-by: Kemal Bektas <[email protected]>

* min_turning_r_ getting param fix (#4510)

* min_turning_r_ getting param fix

Signed-off-by: Ivan Radionov <[email protected]>

* Update nav2_mppi_controller/include/nav2_mppi_controller/motion_models.hpp

Signed-off-by: Steve Macenski <[email protected]>
Signed-off-by: Ivan Radionov <[email protected]>

---------

Signed-off-by: Ivan Radionov <[email protected]>
Signed-off-by: Steve Macenski <[email protected]>
Co-authored-by: Ivan Radionov <[email protected]>
Co-authored-by: Steve Macenski <[email protected]>

* fixing gz sim launch file by using gz directly (#4514)

* port wait behavior to new gazebo (#4471)

Signed-off-by: stevedan <[email protected]>
Signed-off-by: Steve Macenski <[email protected]>
Co-authored-by: Steve Macenski <[email protected]>

* Completely rewritten spin, backup, and drive on heading tests to remove flakiness (#4515)

* port backup behavior to new gazebo

Signed-off-by: stevedan <[email protected]>

* port drive on heading behavior to new gazebo

Signed-off-by: stevedan <[email protected]>

* completely rewritten spin test

* lint

* complete flaky test rewrite

---------

Signed-off-by: stevedan <[email protected]>
Signed-off-by: Steve Macenski <[email protected]>
Co-authored-by: stevedan <[email protected]>

* Return out of map update if frames mismatch. Signed-off-by Joey Yang (#4517)

Signed-off-by: Joey Yang <[email protected]>

* Fix error_code_id (#4522)

Signed-off-by: redvinaa <[email protected]>

* completely shutdown dyn_params_handler_ (s)  (#4521)

* completely shutdown dny_params_handler_ in nav2_amcl

Signed-off-by: GoesM_server <[email protected]>

* completely shutdown dyn_param_handler_ in controller_server

Signed-off-by: goes <[email protected]>

* compeletly shutdown dyn_params_handler in nav2_costmap_2d

Signed-off-by: goes <[email protected]>

* compeletly shutdown dyn_param_handler in nav2_docking

Signed-off-by: goes <[email protected]>

* completely shutdown dyn_params_handler in nav2_velocity_smoother

Signed-off-by: goes <[email protected]>

* compeletly shutdown dyn_param_handler in waypoint_follower

Signed-off-by: goes <[email protected]>

* typo fixed

Signed-off-by: goes <[email protected]>

* graceful-controller & dwb_controller

Signed-off-by: goes <[email protected]>

* mppi-controller

Signed-off-by: goes <[email protected]>

* navfn_planner & regulated_..controller

Signed-off-by: goes <[email protected]>

* rotation_..controller & samc_planners

Signed-off-by: goes <[email protected]>

* A*planner

Signed-off-by: goes <[email protected]>

* code style

Signed-off-by: goes <[email protected]>

* 1

Signed-off-by: goes <[email protected]>

* fixed

Signed-off-by: goes <[email protected]>

* fix the usage of weak_ptr

Signed-off-by: goes <[email protected]>

* code-style

Signed-off-by: goes <[email protected]>

* weak_ptr released

Signed-off-by: goes <[email protected]>

* code style

Signed-off-by: goes <[email protected]>

* code style

Signed-off-by: goes <[email protected]>

* code style

Signed-off-by: goes <[email protected]>

* code style update

Signed-off-by: goes <[email protected]>

* back

Signed-off-by: goes <[email protected]>

* rebase conflict resovled

Signed-off-by: goes <[email protected]>

* rebase error fixed

Signed-off-by: goes <[email protected]>

* fixed2

Signed-off-by: goes <[email protected]>

* rebase fixed 3

Signed-off-by: goes <[email protected]>

* 33

Signed-off-by: goes <[email protected]>

* shared_ptr into weak_ptr

Signed-off-by: GoesM <[email protected]>

* remove adundant node.resest()

Signed-off-by: GoesM <[email protected]>

---------

Signed-off-by: GoesM_server <[email protected]>
Signed-off-by: goes <[email protected]>
Signed-off-by: GoesM <[email protected]>
Co-authored-by: GoesM_server <[email protected]>

* check nullptr in smoothPlan() (#4544)

* check nullptr in smoothPlan()

Signed-off-by: GoesM <[email protected]>

* code-style

Signed-off-by: GoesM <[email protected]>

* code-style

Signed-off-by: GoesM <[email protected]>

* simple change

Signed-off-by: GoesM <[email protected]>

---------

Signed-off-by: GoesM <[email protected]>
Co-authored-by: GoesM <[email protected]>

* check nullPtr in `computeControl()` (#4548)

Signed-off-by: goes <[email protected]>
Co-authored-by: goes <[email protected]>

* Straight analytic expansions (#4549)

* Add test to verify analytic expansions are straight

Signed-off-by: James Ward <[email protected]>

* Use continuous scaling of angle when performing analytic expansion

Only applies to Hybrid A* - behaviour in lattice planner is unchanged

Signed-off-by: James Ward <[email protected]>

---------

Signed-off-by: James Ward <[email protected]>

* Rviz tool to get cost of costmap cell (#4546)

* Added GetCost srv

Signed-off-by: Jatin Patil <[email protected]>

* Added Service in costmap_2d

Signed-off-by: Jatin Patil <[email protected]>

* Added Rviz tool

Signed-off-by: Jatin Patil <[email protected]>

* Fixed Styling

Signed-off-by: Jatin Patil <[email protected]>

* Fixed Styles and Linting

Signed-off-by: Jatin Patil <[email protected]>

* Fixed Linting

Signed-off-by: Jatin Patil <[email protected]>

* Added Bool use_footprint to srv

Signed-off-by: Jatin Patil <[email protected]>

* Added unit test for costmap costcell cost service

Signed-off-by: Jatin Patil <[email protected]>

* Fixed unit test script

Signed-off-by: Jatin Patil <[email protected]>

* Added theta, Updated unit test, Updated rviz tool service call logic

Signed-off-by: Jatin Patil <[email protected]>

* Updated requested changes

Signed-off-by: Jatin Patil <[email protected]>

---------

Signed-off-by: Jatin Patil <[email protected]>

* Switch to new-style static_transform_publisher arguments. (#4563)

These arguments have been the preferred way to use things
since at least Humble.  This avoids warnings when running it for the tests.

Signed-off-by: Chris Lalancette <[email protected]>

* Updated slack link (#4565)

Signed-off-by: Steve Macenski <[email protected]>

* Update README.md (#4589)

Signed-off-by: Steve Macenski <[email protected]>

* fix flickering visualization (#4561)

* Fix Flickering visualization

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Refactoring Costmap2DPublisher and Costmap2DROS

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Refactoring costmap_2d_ros.cpp

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Refactoring Costmap2DPublisher and Costmap2DROS

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Update costmap_2d_publisher.cpp

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Change map_vis_z from float to double

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Add comment to  map_vis_z_ parameter

Signed-off-by: Vladyslav Hrynchak <[email protected]>

---------

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Copy fix-terminate diff from opennav_docking repo (#4598)

* Copy fix-terminate diff from opennav_docking repo

Signed-off-by: redvinaa <[email protected]>

* Lint

Signed-off-by: redvinaa <[email protected]>

---------

Signed-off-by: redvinaa <[email protected]>

* Fix race condition in AMCL for #4537 (#4605)

* Fixed timed_behavior.hpp (#4602)

Signed-off-by: Vladyslav Hrynchak <[email protected]>

* Adding new Nav2 loopback simulator (#4614)

* adding Nav2 loopback sim

Signed-off-by: Steve Macenski <[email protected]>

* drop performance by half

* lintin

* Add multirobot usecase comment

* fixing copy paste error

* fixing review comments

---------

Signed-off-by: Steve Macenski <[email protected]>

* Added laser data from map in nav2_loopback_sim (#4617)

* Added laser data from map

Signed-off-by: Jatin Patil <[email protected]>

* Fixed Linting

Signed-off-by: Jatin Patil <[email protected]>

* Fixed Linting

Signed-off-by: Jatin Patil <[email protected]>

* Fixed few requested changes

Signed-off-by: Jatin Patil <[email protected]>

* Fixed linting

Signed-off-by: Jatin Patil <[email protected]>

* Requested changes

Signed-off-by: Jatin Patil <[email protected]>

* Linting

Signed-off-by: Jatin Patil <[email protected]>

* Added parameters and fixed requested changes

Signed-off-by: Jatin Patil <[email protected]>

* linting

Signed-off-by: Jatin Patil <[email protected]>

* Added scan  using LineIterator

Signed-off-by: Jatin Patil <[email protected]>

* LineIterator max_distance or range_max

Signed-off-by: Jatin Patil <[email protected]>

* min of max_distance or range_max

Signed-off-by: Jatin Patil <[email protected]>

* final updates working correctly

Signed-off-by: Jatin Patil <[email protected]>

* Fix lint

Signed-off-by: Jatin Patil <[email protected]>

* requested changes

Signed-off-by: Jatin Patil <[email protected]>

* Update nav2_loopback_sim/nav2_loopback_sim/loopback_simulator.py

Signed-off-by: Steve Macenski <[email protected]>

* Update nav2_loopback_sim/nav2_loopback_sim/loopback_simulator.py

Signed-off-by: Steve Macenski <[email protected]>

---------

Signed-off-by: Jatin Patil <[email protected]>
Signed-off-by: Steve Macenski <[email protected]>
Co-authored-by: Steve Macenski <[email protected]>

* Making base frame ID for map to base link transform based on base frame ID parameter (#4632)

Signed-off-by: Steve Macenski <[email protected]>

* Update Smac Planner for rounding to closest bin rather than flooring (#4636)

* adding stamped option for loopbacks im (#4637)

Signed-off-by: Steve Macenski <[email protected]>

---------

Signed-off-by: GoesM_server <[email protected]>
Signed-off-by: Alberto Tudela <[email protected]>
Signed-off-by: HovorunBh <[email protected]>
Signed-off-by: Steve Macenski <[email protected]>
Signed-off-by: redvinaa <[email protected]>
Signed-off-by: Kemal Bektas <[email protected]>
Signed-off-by: Ivan Radionov <[email protected]>
Signed-off-by: stevedan <[email protected]>
Signed-off-by: Joey Yang <[email protected]>
Signed-off-by: goes <[email protected]>
Signed-off-by: GoesM <[email protected]>
Signed-off-by: James Ward <[email protected]>
Signed-off-by: Jatin Patil <[email protected]>
Signed-off-by: Chris Lalancette <[email protected]>
Signed-off-by: Vladyslav Hrynchak <[email protected]>
Co-authored-by: GoesM <[email protected]>
Co-authored-by: GoesM_server <[email protected]>
Co-authored-by: Alberto Tudela <[email protected]>
Co-authored-by: Bohdan <[email protected]>
Co-authored-by: Vince Reda <[email protected]>
Co-authored-by: Kemal Bektas <[email protected]>
Co-authored-by: Kemal Bektas <[email protected]>
Co-authored-by: Ivan Radionov <[email protected]>
Co-authored-by: Ivan Radionov <[email protected]>
Co-authored-by: Stevedan Ogochukwu Omodolor <[email protected]>
Co-authored-by: stevedan <[email protected]>
Co-authored-by: Joey Yang <[email protected]>
Co-authored-by: James Ward <[email protected]>
Co-authored-by: Jatin Patil <[email protected]>
Co-authored-by: Chris Lalancette <[email protected]>
Co-authored-by: Vladyslav Hrynchak <[email protected]>
josephduchesne pushed a commit to josephduchesne/navigation2 that referenced this issue Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants