Redesign s3s-fs

[Nugine]

Currently s3s-fs is just an example for how to implement a storage service with s3s.

Current Problems

• logical concurrent safety is not guaranteed
• internal metadata implementation is too naive
• difficult to support more operations than the basic ones

Goals

• a better architecture, correctness first, more extensible
• still an example, easy to understand, hide details behind abstractions

Non-goals

• extremely high performance
• production ready

[Nugine]

meta store: sqlite

blob store: filesystem

object-level rwlock

[balchua]

I'd like to learn more about s3s-fs, i know its not meant for production.
However i'd like to see how to add Http body limit, is this something that can be implemented in the s3s-fs? A good pointer will be great.
I am also new to rust and hyper. 😀. So there's definitely skill issue here.

Btw thanks for the project!!

[Nugine]

However i'd like to see how to add Http body limit, is this something that can be implemented in the s3s-fs? A good pointer will be great.

Http body limit can be implemented as a middleware like axum::extract::DefaultBodyLimit.
s3s is based on hyper so you can easily combine it with hyper ecosystem and tower ecosystem.

To add http body limit to s3s-fs:

1. Add a cli arg for configuring max body limit
2. Wrap the S3Service with a body limit middleware
3. Test whether it works

Or, another way is that we implement stream-aware body limit in s3s.
I'm not sure about the best design right now. It may be related with #155.

Good luck!

[balchua]

Thanks @Nugine , i tried implementing a simple one, but struggling to return a hyper Response, when returning a 413 error.

I was trying to add a layer using the ServiceBuilder. I think im doing something wrong. Was wondering if you happen to have a sample on how to add a middleware that returns an Error.

Thanks again for your help.

Im also interested on implementing the the metadata store as you mentioned above.

[Nugine]

how to add a middleware that returns an Error

The middleware works on Request/Response. You can just convert the error to a Response by setting the status code (413) and corresponding body. S3Error can be also converted to a Response.

Related

• CORS example #164
• Expose Error Serialization To Consumers to Allow Ergnomoic Middleware #157

Im also interested on implementing the the metadata store as you mentioned above.

Thanks! I have no clear design for now. You can try to implement ideas in your fork and show it here.
We may accept a well-designed architecture from contributors. See also #174