Fix bug that can cause early exit from loiter

Fix bug that can cause early exit from loiter #438