Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add chunk bulk upload API #4872

Open
tmm360 opened this issue Oct 21, 2024 · 1 comment
Open

Add chunk bulk upload API #4872

tmm360 opened this issue Oct 21, 2024 · 1 comment

Comments

@tmm360
Copy link
Contributor

tmm360 commented Oct 21, 2024

Summary

Current http and websocket upload APIs are heavily suboptimized. A better API to improve upload speed is necessary.

Motivation

Current POST /chunks API and websocket protocol work in the same way, sending one chunk at time, and waiting ack from node.
All the time the network is always waiting for something, or to receive new chunk data, or confirmation that data has been received. The network line is always empty of data.

Instead, to have a continuous stream of chunks is a really better solution.

Implementation

I've implemented an alternative rest API into an experimental reverse proxy project that I've called "BeeTurbo". It is putted directly in front of bee.

api implementation: https://github.com/Etherna/bee-turbo/blob/3516cf9f0b67af3ae58308d5f7433048fdbff5aa/src/BeeTurbo/Handlers/ChunksBulkUploadHandler.cs

Basically, it receives a set of chunks with a single POST request. Payload is composed as continuous list of chunks, where each one is described as:

  • uint16 with chunk's size in byte
  • chunk's payload, of the given size
  • swarm hash, used only to check chunk's integrity

Drawbacks

The new api is not retro-compatible, a new endpoint has to be introduced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
@tmm360 and others