fluent-plugin-s3 doesn't seem to work with AWS short-term credentials

[sanjay-curium]

Describe the bug

I have been trying to upload aggregated logs through fluentD to an s3 bucket. The entire set-up works if I use AWS long-term access keys but not with the short-term credentials. The error thrown looks like the following.

unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.>"

To Reproduce

1. Build a docker image for the fluentD using the following contents of the dockerfile.

FROM fluentd:latest
USER root
RUN fluent-gem install fluent-plugin-s3
COPY fluentD.conf /fluentd/etc/fluent.conf

2. My fluentD.conflooks like this.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<source>
    @type http
    port 9880
    bind 0.0.0.0
</source>


<match *.*>
  @type s3
  aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}" 
  aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}" 
  aws_sso_key "#{ENV['AWS_SSO_KEY']}"
  s3_bucket "#{ENV['BUCKET_NAME']}" 
  s3_region "#{ENV['AWS_REGION']}" 

  <buffer tag,time>
    @type file
    path /output/test.log
    timekey 60 # 1 minute partition
    timekey_wait 10s
    timekey_use_utc true
    chunk_limit_size 256m
    append true
  </buffer>

</match>

3. Run the container using the following command.

docker run -p 9880:9880 -p 24224:24224 -v ./logs:/output/ -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_SSO_KEY=$AWS_SESSION_TOKEN -e BUCKET_NAME=$BUCKET_NAME -e AWS_REGION=$AWS_REGION image_name:tag

Expected behavior

Logs show up on the s3 bucket if I use AWS long-term credentials but not with the short-term ones.

Your Environment

- Fluentd version: gem 'fluentd' version '1.16.0' -- `fluentd:latest` docker image.
- s3 plugin version: gem 'fluent-plugin-s3' version '1.7.2'

Your Configuration

@type forward port 24224 bind 0.0.0.0 @type http port 9880 bind 0.0.0.0

<match .>
@type s3
aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
aws_sso_key "#{ENV['AWS_SSO_KEY']}"
s3_bucket "#{ENV['BUCKET_NAME']}"
s3_region "#{ENV['AWS_REGION']}"

<buffer tag,time>
@type file
path /output/test.log
timekey 60 # 1 minute partition
timekey_wait 10s
timekey_use_utc true
chunk_limit_size 256m
append true

Your Error Log

2023-08-02 04:28:06 +0000 [error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.>"

Additional context

No response