You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
generate_status_report can take hours on a large site and it runs 4 reports which are each expensive in different ways.
I'm fairly certain that all of these reports can be refactored one by one to to all run in a delta mode. Each of the reports is saved so we know the previous results from when it last ran. We should be able to run stats on only the files which have changed since the last run time and append them. Files which have been deleted we can keep in the tool_objectfs_objects marked as deleted so we can properly track removals, and we can only clean them out of that table via delete_orphaned_object_metadata when for records which we know have been processed by generate_status_report.
In a round about way this should also fix #556 / #411
The text was updated successfully, but these errors were encountered:
brendanheywood
changed the title
Make the generate_status_report work using deltas to make it many order of magnitude faster
Make the generate_status_report work using deltas to make it many orders of magnitude faster
Dec 5, 2023
Hi Brendan :) How are you? Do you reckon this one could be progressed sometime soon or your focus needs to be elsewhere? Not chasing, just checking. We could increase temp table memory allocation for affected clients but your solution sounds way better
generate_status_report can take hours on a large site and it runs 4 reports which are each expensive in different ways.
I'm fairly certain that all of these reports can be refactored one by one to to all run in a delta mode. Each of the reports is saved so we know the previous results from when it last ran. We should be able to run stats on only the files which have changed since the last run time and append them. Files which have been deleted we can keep in the tool_objectfs_objects marked as deleted so we can properly track removals, and we can only clean them out of that table via delete_orphaned_object_metadata when for records which we know have been processed by generate_status_report.
In a round about way this should also fix #556 / #411
The text was updated successfully, but these errors were encountered: