Workflows intermittently exit prematurely with false success

[sthomson-wyn]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

2.322.0

Deployment Method

ArgoCD

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

We can't reliably reproduce. This happens intermittently. There is possibly a correlation with high load, but we can't confirm it.

Describe the bug

Occasionally, a GHA step will exit before completing, but the step will still be marked as a success, and the workflow will continue (when it shouldn't).

This happens regardless of if it is a shell step, a node step, if set -e is set or not, there appears to be no correlation with any specific step. Sometimes these steps do actually fail, and the self hosted runners do correctly indicate their failure, halting execution, but occasionally not.

Describe the expected behavior

Either the step should not exit prematurely, or the step should fail.

Additional Context

No additional context

Controller Logs

https://gist.github.com/sthomson-wyn/a1f96de687d6020771da00e1894fcf97

Runner Pod Logs

https://gist.github.com/sthomson-wyn/6d8f19c8da1f3be77ad8767e7a384a15