Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster large collections #117

Merged
merged 2 commits into from
Sep 8, 2024
Merged

Faster large collections #117

merged 2 commits into from
Sep 8, 2024

Conversation

SmilyOrg
Copy link
Owner

@SmilyOrg SmilyOrg commented Sep 8, 2024

Fixes a few smaller and bigger issues when it comes to performance loading 100k+ file collections.

Improvements

Tests with collections made from 10M+ dummy files reveal the following improvements. Numbers based on an older i7-5820K 6-Core desktop CPU.

  • 10M files in 1 dir
    • Before: loading starts after 15 seconds
    • After: loading starts in under 1 second
  • 10M files in 3000 dirs
    • Before: loading starts after 38 seconds
    • After: loading starts in under 1 second

Changes

  • Fix the loading spinner only showing the number without "files"
  • Add skip_collection_counts to avoid showing collection counts on startup, which can speed it up in case of large collections
  • Refactor the way collection loading works to avoid some slow operations
  • Fixed a bug where collections with dirs that are a prefix of other dirs would in some cases erroneously also list files in these other dirs (e.g. /vacation would include /vacation2 or /vacation-eu)

Limitations / Breaking changes

This change introduces a limitation of max 50000 directories that can be sourced in a single collection. Considering the previously long loading time of such collections, this is probably not a big concern. If you hit this limitation, please let me know!

@SmilyOrg SmilyOrg merged commit 637fe90 into main Sep 8, 2024
1 check passed
@SmilyOrg SmilyOrg deleted the faster-large-collections branch September 8, 2024 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

panic: determinant of affine transformation matrix is zero
1 participant