Optimizing S3 Bulk Delete Operations on Storj

[ferristocrat]

Background

What is the problem/pain point?

The S3 compatible gateway offered by Storj is capable of handling multi-object delete requests. However, due to the Storj Satellite protocol’s lack of direct support for such operations, the gateway has to translate these bulk delete actions into numerous simultaneous individual delete requests. While the parallelization on the S3 side mitigates performance issues, it inadvertently triggers rate limits and results in internal inefficiencies, as our infrastructure needs to process a high volume of requests at once.

Who is impacted?

The immediate impact is on our internal resources and systems due to the excessive load from numerous simultaneous requests. Additionally, customers are indirectly affected as they experience issues related to rate limits being hit.

What is the impact?

The triggering of rate limits can disrupt user operations, potentially leading to failed delete operations and a diminished user experience. Internally, our systems are strained due to the resource consumption needed to handle a large number of simultaneous requests, potentially leading to reduced overall system performance and stability.

Why now?

Addressing this issue now is crucial as it directly affects the stability and efficiency of our services. As customer usage scales, the inefficiencies and potential for hitting rate limits will only increase, making it imperative to optimize this aspect of our service.

Requirements

User Story

As a customer using the Storj S3 compatible gateway, I want the system to efficiently handle multi-object delete requests without running into rate limits, so that I can ensure reliable and smooth deletion operations without disruptions.

Acceptance Criteria

1. The Storj Satellite should be able to natively handle multi-object delete requests.
2. Implementation of this feature should result in a noticeable reduction in internal resource consumption for delete operations.
3. Rate limits should no longer be triggered during bulk delete operations, ensuring a smooth user experience.

Success Metrics

1. A reduction in the internal resources required to process bulk delete operations.
2. No instances of rate limits being triggered due to delete operations.
3. Positive feedback from the operations team and potentially customers regarding the stability and efficiency of delete operations.

Additional Information

Blueprint that has been created: https://review.dev.storj.io/c/storj/storj/+/11352

Productboard Link

• https://storj.productboard.com/entity-detail/features/22460240

[NiaStorj]

As of 11/25/24:

Still in progress