Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement custom caching strategy #1

Open
polyrand opened this issue Dec 14, 2022 · 0 comments
Open

Implement custom caching strategy #1

polyrand opened this issue Dec 14, 2022 · 0 comments

Comments

@polyrand
Copy link
Contributor

I'm not happy very with the caching strategies that fsspec provides. fsspec optimizes for certain memory access patterns that, I believe, are not always optimal for a DB file.

The mmap caching caught my attention, but the tests I've run are very slow. The mmap logic can be optimized.

I think I should implement my own caching strategy. Some ideas:

  • Least-Frequently-Used strategy: This could be useful for DB files that can't/shouldn't be fully copied to disk.
  • Incremental full mmap. mmap-ed file, eventually the full database. (Have a mmaped bitset to store which pages have been fetched already?).

Both approaches would need to enable sharing a cache between multiple processes in the same VM.

@polyrand polyrand changed the title Custom caching strategy Implement custom caching strategy Dec 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant