You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm not happy very with the caching strategies that fsspec provides. fsspec optimizes for certain memory access patterns that, I believe, are not always optimal for a DB file.
The mmap caching caught my attention, but the tests I've run are very slow. The mmap logic can be optimized.
I think I should implement my own caching strategy. Some ideas:
Least-Frequently-Used strategy: This could be useful for DB files that can't/shouldn't be fully copied to disk.
Incremental full mmap. mmap-ed file, eventually the full database. (Have a mmaped bitset to store which pages have been fetched already?).
Both approaches would need to enable sharing a cache between multiple processes in the same VM.
The text was updated successfully, but these errors were encountered:
polyrand
changed the title
Custom caching strategy
Implement custom caching strategy
Dec 14, 2022
I'm not happy very with the caching strategies that
fsspec
provides.fsspec
optimizes for certain memory access patterns that, I believe, are not always optimal for a DB file.The
mmap
caching caught my attention, but the tests I've run are very slow. Themmap
logic can be optimized.I think I should implement my own caching strategy. Some ideas:
Both approaches would need to enable sharing a cache between multiple processes in the same VM.
The text was updated successfully, but these errors were encountered: