Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the generate_status_report work using deltas to make it many orders of magnitude faster #572

Open
brendanheywood opened this issue Aug 1, 2023 · 2 comments

Comments

@brendanheywood
Copy link
Contributor

brendanheywood commented Aug 1, 2023

generate_status_report can take hours on a large site and it runs 4 reports which are each expensive in different ways.

I'm fairly certain that all of these reports can be refactored one by one to to all run in a delta mode. Each of the reports is saved so we know the previous results from when it last ran. We should be able to run stats on only the files which have changed since the last run time and append them. Files which have been deleted we can keep in the tool_objectfs_objects marked as deleted so we can properly track removals, and we can only clean them out of that table via delete_orphaned_object_metadata when for records which we know have been processed by generate_status_report.

In a round about way this should also fix #556 / #411

@brendanheywood brendanheywood changed the title Make the generate_status_report work using deltas to make it many order of magnitude faster Make the generate_status_report work using deltas to make it many orders of magnitude faster Dec 5, 2023
@anton-catalyst-eu
Copy link

Hi Brendan :) How are you? Do you reckon this one could be progressed sometime soon or your focus needs to be elsewhere? Not chasing, just checking. We could increase temp table memory allocation for affected clients but your solution sounds way better

@brendanheywood
Copy link
Contributor Author

@JoshAudette had some success here #596 on this front

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants