Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

quotas: improve performance of quota updater #193

Open
mdonadoni opened this issue Jan 16, 2023 · 1 comment
Open

quotas: improve performance of quota updater #193

mdonadoni opened this issue Jan 16, 2023 · 1 comment
Labels
priority/longterm size/l system/scalability type/enhancement New feature or request

Comments

@mdonadoni
Copy link
Member

The periodic quota updater takes a long time to complete when dealing with many workflows and/or with big workspaces.

These are some possible improvements:

  1. Currently, Session.commit() is called once for every workflow in store_workflow_disk_quota (here and here). Instead, it should be called only once after all the quota updates have been calculated. This same improvements can also be applied to the other utility functions that are used to update the disk/cpu quotas of workflows and users. In particular, I have noticed that Session.commit() becomes slower the more workflows are loaded from the database.
  2. The disk quota usage of all the workflows are recalculated, even though many workflows have not changed since the last execution of the quota updater. One possible solution to this would be to consider only workflows that are "dirty", that is workflows whose workspace might have changed since the last quota update (e.g. a session has been opened, a file has been deleted/uploaded, etc.)
mdonadoni added a commit to mdonadoni/reana-db that referenced this issue Aug 25, 2023
@mdonadoni
quota: improve performance of quota updater
3e11213 
Improve the performance of the quota updater by "expunging" all
workflows before starting to update quotas, as the Workflow table does
not need to be modified, thus making commits much faster. Also refactor
the various update functions to make their behaviour consistent with
each other.

Partially addresses reanahub#193
@mdonadoni mdonadoni mentioned this issue Aug 25, 2023
Merged
mdonadoni added a commit to mdonadoni/reana-db that referenced this issue Aug 30, 2023
@mdonadoni
quota: improve performance of quota updater
6084fa5 
Improve the performance of the quota updater by "expunging" all
workflows before starting to update quotas, as the Workflow table does
not need to be modified, thus making commits much faster. Also refactor
the various update functions to make their behaviour consistent with
each other.

Partially addresses reanahub#193
mdonadoni added a commit to mdonadoni/reana-db that referenced this issue Sep 1, 2023
@mdonadoni
quota: improve performance of quota updater
8cc26f7 
Improve the performance of the quota updater by "expunging" all
workflows before starting to update quotas, as the Workflow table does
not need to be modified, thus making commits much faster. Also refactor
the various update functions to make their behaviour consistent with
each other.

Partially addresses reanahub#193
mdonadoni added a commit to mdonadoni/reana-db that referenced this issue Sep 1, 2023
@mdonadoni
quota: avoid accessing disk when updating user disk quotas
64e23a4 
Partially addresses reanahub#193
@mdonadoni mdonadoni mentioned this issue Sep 1, 2023
Merged
@mdonadoni
Copy link
Member Author

See also #200 (comment)

mdonadoni added a commit to mdonadoni/reana-db that referenced this issue Sep 5, 2023
@mdonadoni
quota: avoid accessing disk when updating user disk quotas
89c3e0e 
Partially addresses reanahub#193
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/longterm size/l system/scalability type/enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mdonadoni