Add 'hash_pathing_fallback' option, remove File.get_path() method #1017
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Your checklist for this pull request
What is the current behaviour?
We found that
hash_pathing
is useful also for some object storage solutions like MinIO.MinIO maps
/
prefixes into actual directories on the file system. Even if we use recommended file system that allows to store lots of files in single directory, various operations using object listing (including internal ones like MinIO scanner) are significantly slowed down.Unfortunately we can't smoothly migrate to the new naming scheme without fallback.
What is the new behaviour?
hash_pathing_fallback
option. WhenFile.open()
fails to open file using currenthash_pathing
setting, it tries to use opposite one. It works only for gathering file contents, all new files will be stored using the chosen scheme only.File.get_path()
. Check your plugins before applying this change._calculate_path()
method toget_or_create()
method. Path retrieval should not make any changes on the storage.Test plan
Manually checked if fallback actually works. Other cases should be covered by automated tests.