Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'm trying to work with AWS requester pays buckets, but getting an error:

GenericError: Generic S3 error: Error performing list request: Client error with status 403 Forbidden: 

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>There were headers present in the request which were not signed</Message>
    <HeadersNotSigned>x-amz-request-payer</HeadersNotSigned>
    <RequestId>QEA9RD5BETE6BN7E</RequestId>
    <HostId>tfgEYvVBmWEx5EZljTRXuR6S6ocChjas5qF8eXThc0yt6bRyGEJCYjCFJ+/YOvA2Q2DT/NSqBTw=</HostId>
</Error>

The following is a Python example from obstore. It should be easy to interpret what the underlying Rust code is, but I can provide a pure-Rust example if necessary

import obstore
from obstore.store import ClientOptions, S3Store
import boto3

# boto3.Session reads credentials from disk, which then passes them to AmazonS3Builder
session = boto3.Session()

# Set x-amz-request-payer as a default header for every request
options = ClientOptions(default_headers={b"x-amz-request-payer": "requester"})

# This is a bucket exposed through the AWS open data registry
# https://registry.opendata.aws/naip/
# It's public with requester-pays turned on
store = S3Store.from_session(session, bucket="naip-visualization", client_options=options)

# Make the request
result = obstore.list(store, "ny/2022/60cm/rgb/40073/").collect()

Describe the solution you'd like

Include AWS header in signature calculation. Presumably there would need to be some way to tell object_store which default headers to include in its signing mechanism?

Describe alternatives you've considered

Additional context

This AWS docs page has a reference on requester pays and request signing

The only prior discussion I can find is on discord:

Hi everyone, if anyone is familiar with the object_store implementation in arrow-rs I had a question; is there a way to pass in arbitrary AWS (specifically S3) options via HTTP headers? For example, I don't see implicit support for requester pays buckets, which is hinted at by passing in this header value pair: x-amz-request-payer : requester so I'm wondering if there's a way to add in our own options like that.

Not on a per-request basis, as it sort of breaks the abstraction of being any object store, but you could look at overriding the headers using ClientOptions when you build the AmazonS3 client