Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental brick server alternative with on-disk metadata DB #22

Open
tatsuya6502 opened this issue May 14, 2014 · 0 comments
Open

Experimental brick server alternative with on-disk metadata DB #22

tatsuya6502 opened this issue May 14, 2014 · 0 comments
Assignees
Milestone

Comments

@tatsuya6502
Copy link
Member

Create an experimental, alternative brick server module of brick_ets that does not use an in-memory ETS table (ctab) to store metadata part of key-values but uses on-disk (HyperLevelDB) metadata DB introduced by GH17 redesigning disk storage.

The current brick_ets only reads from HyperLevelDB during start up and stores copies of all keys in in-memory ETS table, but this alternative module will not use the ETS table and read from HyperLevelDB for each read request from a client. It may cache some frequently used metadata on a 2Q cache (a refinement of LRU cache with scan resistant) available as Erlang NIF: arekinath/e2qc.

I have a project that requires low memory footprint and does not demand super low read latency nor in-memory key scan. A couple of months ago, I tried to load 50 million key-values to single-node Hibari but top command showed it took 9.149 GB for RES memory, which was not acceptable. One reason for that was the keys were big (53 bytes), so I will need to unload all or majority of metadata from the memory.

I believe an ultimate goal of redesigning disk storage will be having tired storage per Hibari table because we can use micro transaction within a brick. But for now, I want to quickly put together on-disk metadata storage based brick server module.

@tatsuya6502 tatsuya6502 self-assigned this May 14, 2014
@tatsuya6502 tatsuya6502 added this to the v0.3 milestone Jun 9, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant