Ephemeral runners randomly failing on DELETE request

[velkovb]

We run a total of 24 scale sets - 12 different types and we run them in amd64 and arm64 flavours. I have noticed that some of our amd64 ephemeral runners end up in a failed state. After digging into the issue I saw the following error message in the logs:
2024-10-29T08:16:43.865368226Z stdout F [RUNNER 2024-10-29 08:16:43Z ERR GitHubActionsService] DELETE request to https://pipelinesghubeus21.actions.githubusercontent.com/ZGECcFA0iBFQiiokBtxcNrTLeYVlRtiBGMkAobx3ZHH0TBc8tF/_apis/distributedtask/pools/1/sessions/893bf65f-341e-442d-9790-0b075e79c041 failed. HTTP Status: Forbidden

Seems to be happening on the arm64 ones as well be they never reach a failed state (5 failed attempts). Couldn't find any information on this error and what might be causing it.

The problems is that after they reach failed state we have to clean them up manually or it has even happened when the runner has a job assigned.

[velkovb]

This was happening only when we initially did the setup and not being reproduced. Not idea what caused it.