New Concurrency Model

[nvoxland]

🚀 🚀 Pull Request

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

Replaces the existing lock-based write consistency system with a new commit-log style system inspired by Delta Lake.

With this system, there is a now a new _deeplake_log directory which store atomically created files describing the operations performed against deeplake. Files in the root of that directory correspond to changes to the "main" branch, and sub-directories contain operations for different branches.

Example directory structure:

_deeplake_log
    _meta
        00000000000000000001.json
        00000000000000000002.json
        00000000000000000003.json
        00000000000000000004.json
    65af17c7c4bc4cbbbcafa
        00000000000000000001.json
        00000000000000000002.json
    e8d4ce5b70e8476fb811
        00000000000000000002.json

With an example json file of:

{"protocol":{"minReaderVersion":4,"minWriterVersion":4}}{"metadata":{"createdTime":1696350362244,"description":null,"id":"0d1a97d8-6cc4-4e12-b603-b5af37e8024d","name":null}}{"branch":{"fromId":"","fromVersion":-1,"id":"","name":"main"}}

Readers are always working from a particular branch + version based on the file name, such as e8d4ce5b70e8476fb811 + 2 and build up the current state by union-ing all the json files. All files (both deep log files plus data files) are immutable, with new versions specifying new file versions rather than overwriting them.

For optimization purposes, the unioned json files are periodically checkpoint()ed into parquet files, and unused files can be vacuumed() away.

See https://hubdb.slack.com/archives/C01HCMDL97F for more information

Things to be aware of

In progress...

Things to worry about

In progress...

Additional Context

In progress...