Chunk the blacklists and watchlist to reduce regex recompiles upon reloading the blacklists and watchlist #13765
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR does:
Rule
s into twoRule
s with the bulk of each list of entries in one of the two and up to 100 entries in the second. This results in dramatically reducing the computation time expended upon a watchlist or blacklist change for recompiling the regexes used. Overall, it should reduce the computation time spent on that task by an estimated 95%.!!/scan-time
and!!/scan-force-time
, which add an elapsed time to the output of the!!/scan
command showing how long it took to execute the scan(s).Testing for this begins here with the times for the final version tested starting here and a baseline without the chunk/split of the BL/WL
Rule
s after that, starting here. The notable difference being the time after the!!/watch opjd(?<!d)
command, which is 14.439 seconds in the baseline, but 4.641 seconds with these changes. The 4.641 seconds is only slightly higher than the 4.491 to 4.619 seconds required for the scan when there hasn't been a change that needs at least one of the regular expressions recompiled.