Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"OperationalError: could not lock RRD" after upgrading from python3-rrdtool from 1.7.0 to 1.7.2 #1228

Open
ekondrashev opened this issue May 22, 2023 · 1 comment

Comments

@ekondrashev
Copy link

ekondrashev commented May 22, 2023

Describe the bug
Hi RRD Tool Community!

After upgrading of python3-rrdtool from 1.7.0 to 1.7.2 we started seeing below stacktraces during our performance tests cycles

May 14 01:54:03 django-admin[544208]: Traceback (most recent call last):
May 14 01:54:03 django-admin[544208]:   File "debian/abc/usr/lib/python3.10/dist-packages/abc/models.py", line 519, in a
May 14 01:54:03 django-admin[544208]:   File "debian/abc/usr/lib/python3.10/dist-packages/abc/results/ts.py", line 55, in update
May 14 01:54:03 django-admin[544208]:   File "debian/abc/usr/lib/python3.10/dist-packages/abc/results/ts.py", line 76, in update_rrd
May 14 01:54:03 django-admin[544208]:   File "debian/abc/usr/lib/python3.10/dist-packages/abc/results/models.py", line 695, in update
May 14 01:54:03 django-admin[544208]:   File "debian/abcn/usr/lib/python3.10/dist-packages/abc/r/rrd.py", line 100, in update
May 14 01:54:03 django-admin[544208]: rrdtool.OperationalError: could not lock RRD

in the rrd.update we just call

rrdtool.update([f] + args)

where

 f=“rrdfile123”
args=[‘--’, ‘1684022:xxx.yyyy:0.0:0:0.zzz:0.qqq:0.hhh:0.ggg:0:U:0:U:0:0:tttt:0’, ‘1684023:xxx.yyyy:0.0:0:0.zzz:0.qqq:0.hhh:0.ggg:0:U:0:U:0:0:tttt:1’]

To Reproduce

  • call rrdtool.update from multiple processes
    • use different filename in each process

I barely can estimate the amount of processes that were calling rrdtool.update, but let me state that during test period there were 20000 calls of that method and test started on May 13 18:03:14 and finished on May 14 21:56:06, and there is a limit of having 20 workers at the same time.
I was looking into the logs and can confirm there were no rrdtool.update calls done at the same time to the same file.

So the question is whether there were any changes done to the tool that should be taken into consideration?
I was looking into change log file, but found nothing related to that
https://github.com/oetiker/rrdtool-1.x/blob/v1.7.2/CHANGES

Also was trying to analyze the changes itself, but that also gave no much results
https://github.com/oetiker/rrdtool-1.x/compare/v1.7.0..v1.7.2

Expected behavior
No OperationalError occur

Additional context

It is important to note that when we where using 1.7.0, it was executed on Ubuntu 18.04 environment, while now 1.7.2 is being used on Ubuntu 22.04.

@mnalis
Copy link

mnalis commented Jul 26, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants