Sync file system using nats object and kv store

[gedw99]

I am looking for help in how to do this with nats.

1st Problem domain:
Clients have files, but are just caches. The files are text files that our golang code knows how to parse.

Client makes a tiny one line changes to file.

Clients want to send each changes to NATS and have those changes merged on a server ( the master of the file ), and also to all clients that have the same file.

2nd problem domain is binary files such as images and fonts.

i think only full sync on this is possible, using hash of the file.

---

Proposed Solution:

Client create a one line patch and send them to NATS KV.

The Sync golang process consumes the patch and updates the master files.

I would love to use NATS object store, but worried about perf / efficiency. I would have to pull the data out of the NATS object store, do the patch and replace the NATS object store, because there is no Patch function for NATS Object store.

The Sync process also puts the patch onto a queue so that all other clients can get the patch.

Clients confuse the patch message and runs the same merge golang code as the server , merging the change into their local file cache.

is this a decent approach?

Is there something you think is missing or ever-looked ?

I assume that Clients can't do offline edits can happen, because then that will lead to merge faults.

thanks for reading this !!

[derekcollison]

You could do that, we are looking into adding in patch like behavior to objectstore but since its code and you can compress it, even large files will be pretty small compressed..

[gedw99]

Thanks @derekcollison

I ended up using what got uses . Unified diff for text base files. It’s universal and fast to git diff , patch and merge.

https://www.gnu.org/software/diffutils/manual/html_node/Unified-Format.html

The implementation is here . thanks golang team !!

https://github.com/golang/tools/tree/master/internal/diff

—-

for images I use a tiling based approach with the images as smaller images and the kv store as the index to know what resolution you want back out .

—-

for binary executables I used a simple binary aware diff path merge.

I know there is ADR discussion about how nats storage can be used as more as a searchable system. It would be interesting but you will always need a way then to also do a migration of the data also .