Merge the small read io

[WenyXu]

          Too many small requests could result in an expensive bill; there is an optimization we can do in the future(maybe not in this PR) for object stores like s3. If the `ranges` are almost continuous, we can merge these ranges into a large chunk and fetch this chunk in the preferred size concurrently.

Originally posted by @WenyXu in #2959 (comment)

[WenyXu]

See also:

• support merge io read api by settings apache/opendal#3675
• https://docs.rs/object_store/latest/src/object_store/lib.rs.html#581-588
• https://github.com/datafuselabs/databend/blob/a98335d33e7abfd34189e7f32c06ab34d53c64d0/src/query/storages/fuse/src/io/read/block/block_reader_merge_io_async.rs

[L-Fiori]

Hello, I'm new to the project and I would like to contribute. Not sure if this is a good first issue though.

As I understand it, you're concurrently fetching data from Parquet files using a function that receives a vector of ranges, and your concern is that too many small requests can lead to expensive bills, so an optimization would be to merge ranges that are close together before fetching data, is that right? In this case, what would be a reasonable distance between the ranges? As I am not familiar with the kind of data that is being fetched.

[WenyXu]

Hello, I'm new to the project and I would like to contribute. Not sure if this is a good first issue though.

As I understand it, you're concurrently fetching data from Parquet files using a function that receives a vector of ranges, and your concern is that too many small requests can lead to expensive bills, so an optimization would be to merge ranges that are close together before fetching data, is that right? In this case, what would be a reasonable distance between the ranges? As I am not familiar with the kind of data that is being fetched.

Thanks @L-Fiori. My bad; my colleague @QuenKar is working on this. I forget to update this issue.

In this case, what would be a reasonable distance between the ranges?

This is key to this issue, and my colleague is doing some benchmarking to figure it out. We will use these benchmark results to select an optimized range distance(and the benchmark results may be posted in related PRs).

[L-Fiori]

Interesting! I'll stay tuned for other issues I might want to tackle, thanks for the reply ;)

[tisonkun]

@WenyXu This issue seems stale. Do we have other updates?

Or @L-Fiori maybe you can just take over this issue and submit a patch >_<

[WenyXu]

@WenyXu This issue seems stale. Do we have other updates?

We still send a lot of small IO requests.