Chunked objects, distributed chunks

[andrewhodel]

If an object is large, chunk the data when sending it and allow chunks to be distributed by different nodes by chunking in json.

Sender
For each 1mb of object size
Send object that has extra field of full object count and extra field of original object hash in looping order (B start at 0, C start at 0+3, D start at 0+6) considering that it is being sent to 3 different nodes (B, C and D).

Receiver
Once all of full object count are received, recombine full object and remove parts

This results in nodes B, C and D getting different chunks and they can send them to each other while that effect cascades through the network.

The problem is that the parts could be removed in a diff from a node that already received and removed them.

The fix is to never remove object parts in diff and only remove them if the full object does not exist and there are no parts for it in the diff.