Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'hash_pathing_fallback' option, remove File.get_path() method #1017

Merged
merged 1 commit into from
Jan 13, 2025

Conversation

psrok1
Copy link
Member

@psrok1 psrok1 commented Jan 13, 2025

Your checklist for this pull request

  • I've read the contributing guideline.
  • I've tested my changes by building and running the project, and testing changed functionality (if applicable)
  • I've added automated tests for my change (if applicable, optional)
  • I've updated documentation to reflect my change (if applicable)

What is the current behaviour?

We found that hash_pathing is useful also for some object storage solutions like MinIO.

MinIO maps / prefixes into actual directories on the file system. Even if we use recommended file system that allows to store lots of files in single directory, various operations using object listing (including internal ones like MinIO scanner) are significantly slowed down.

Unfortunately we can't smoothly migrate to the new naming scheme without fallback.

What is the new behaviour?

  • Add hash_pathing_fallback option. When File.open() fails to open file using current hash_pathing setting, it tries to use opposite one. It works only for gathering file contents, all new files will be stored using the chosen scheme only.
  • Removed deprecated File.get_path(). Check your plugins before applying this change.
  • Moved directory creation from _calculate_path() method to get_or_create() method. Path retrieval should not make any changes on the storage.

Test plan

Manually checked if fallback actually works. Other cases should be covered by automated tests.

@psrok1 psrok1 merged commit 4113e72 into master Jan 13, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant