Improve performance of `analyze` #83

greezybacon · 2019-01-30T23:05:56Z

Overall, this results in about a 20% reduction in the CPU time of the analyze phase of the sync.

This patch avoid resorting the directory tree after the sync. Instead, it sorts the new items pulled during the sync and merges them into the new tree without sorting both lists again.

This changes the function that converts the rsync listing to the tree state files. In my tests, it reduces the time on a list of 250k files from 12 seconds down to 7.

This significantly reduces the processor load in stripping the leading 12 chars from the tree state files. On my test file of 250k files, time is reduced from 960ms to 140ms. Since the `strip_mode` function is used four times on a normal run, this change saves 3s or processor time from my test files.

Jared Hancock added 3 commits January 30, 2019 16:04

perf: Avoid sorting the entire tree at the end

17e3aa6

This patch avoid resorting the directory tree after the sync. Instead, it sorts the new items pulled during the sync and merges them into the new tree without sorting both lists again.

perf: Better regex for scrubbing rsync listing

24695ef

This changes the function that converts the rsync listing to the tree state files. In my tests, it reduces the time on a list of 250k files from 12 seconds down to 7.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve performance of `analyze` #83

Improve performance of `analyze` #83

greezybacon commented Jan 30, 2019

Improve performance of analyze #83

Are you sure you want to change the base?

Improve performance of analyze #83

Conversation

greezybacon commented Jan 30, 2019

Improve performance of `analyze` #83

Improve performance of `analyze` #83