Intermittent logging issues in Papertrail

[giovannif23]

A couple weeks ago we noticed logs stopped flowing from our EKS nodeless clusters to Solarwinds. We restarted the rkubelog pod and it began reporting back to Solarwinds. 2 hours later reporting stopped again, restarting the pod again solved the problem in the short terms. Fast forward to this morning and the pods stopped reporting logs, 2 hours later they started reporting logs again without doing anything on our end. We continue to have intermittent logging issues. We saw both issues on two environments. Restarting has stopped the logging issue in one environment, however, we are still having issues in another. We have restarted several times over the past week and it will stop log flowing after a day or so.

[girishranganathan]

Hello @giovannif23, thx for reporting this issue.
Can you please tell me which version of rkubelog are you using?

[giovannif23]

@girishranganathan we are using r17.

[girishranganathan]

Thx @giovannif23. I will try to investigate this on my end. While I investigate this further: can you please give this image a try: quay.io/solarwinds/rkubelog:r19-b2 ?

[giovannif23]

@girishranganathan I'll give it a try and I'll let you know. Thank you.

[girishranganathan]

@giovannif23 can you please try this new image: quay.io/solarwinds/rkubelog:r19rc2? The earlier has the same issue you had reported. I have tried to fix the issue in this new version + i tested this new image on an mid-sized eks cluster for almost a week.

[cfroystad]

@girishranganathan we're still seeing this issue with rkubelog:r19rc2. I've been unable to identify any pattern to when or why it happens, but a simple restart always fixes the problem. The environment in question is on kubernetes 1.19 if that's useful information.

[vovxox]

having the exact same issue with rkubelog:r19-b2. is there any fix available?

[on2air]

also having same issue. I have to restart rkubelog pod every 3-4 days for it to reset and start logging again. Please investigate. When logging stops we no longer can get notified of issues so instability is causing concern.

[on2air]

any chance this get fixed? such a pain...

[hholst80]

This is still a problem on the critical path.

I will have to move quickly. We do require to have logging working by the end of August. What can we do?

Please escalate within papertrail R&D!

[ingin97]

We are still seeing this issue. Any fix to this issue yet?