Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Authentication error using IAM role #146

Open
1 of 2 tasks
serapch opened this issue Oct 18, 2024 · 2 comments
Open
1 of 2 tasks

Authentication error using IAM role #146

serapch opened this issue Oct 18, 2024 · 2 comments

Comments

@serapch
Copy link

serapch commented Oct 18, 2024

(check apply)

  • read the contribution guideline
  • (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

The issue

We're migrating from using Elasticsearch to Opensearch, both hosted in AWS, but the authentication of fluentd with assumed IAM role to Opensearch service fails with the following error:

missing credentials, provide credentials with one of the following options: (Aws::Sigv4::Errors::MissingCredentialsError): :access_key_id and :secret_access_key, :credentials, :credentials_provider

The used role from the EC2 node where fluentd is running has policies attached (despite it's said in the documentation that ) to it including AmazonOpenSearchServiceFullAccess.

We used almost the same configuration for sending data to Elasticsearch but there we were using basic authentication (setting hosts, port, scheme, ssl_version, username and password) and there weren't any issues.

I looked up in discuss.kubernetes.io and in your project open&closed issues but I couldn't find any with the same error thrown.

Steps to replicate

Fluentd configuration
data:
  01_sources.conf: |-
    # Used for health checking
    <source>
      @type http
      port 9880
      bind 0.0.0.0
    </source>

    # Emits internal metrics to every minute, and also exposes them on port
    # 24220. Useful for determining if an output plugin is retryring/erroring,
    # or determining the buffer queue length.
    <source>
      @type monitor_agent
      bind 0.0.0.0
      port 24220
      tag fluentd.monitor.metrics
    </source>

    <system>
      root_dir /fluentd/
      log_level debug
    </system>

    <source>
      @type http
      @label @SAG
      port 8443
      bind 0.0.0.0
      <parse>
        @type multi_format
        <pattern>
          format json
          time_type string
          time_format %Y-%m-%dT%H:%M:%S.%L%z
        </pattern>
        <pattern>
          format json
          time_type string
          time_format %Y-%m-%dT%H:%M:%S.%L
        </pattern>
        <pattern>
          format none
          message_key log
        </pattern>
      </parse>
    </source>
  02_filters.conf: '# Filters'
  03_dispatch.conf: '# Dispatch'
  04_outputs.conf: |-
    <label @SAG>
      <filter **>
        @type prometheus
        <metric>
          name fluentd_input_status_num_records_total
          type counter
          desc The total number of incoming records
          <labels>
            tag ${tag}
            hostname ${hostname}
            REGION ${REGION}
            ACCOUNT_ALIAS ${ACCOUNT_ALIAS}
            CLUSTER_NAME ${CLUSTER_NAME}
          </labels>
        </metric>
      </filter>
      <match **>
        @type copy
        <store>
          @id "aws-opensearch-service"
          @type "opensearch"
          @log_level debug
          include_tag_key false
          tag_key tag
          <endpoint>
            url "https://#{ENV['OUTPUT_HOST']}"
            region eu-west-1
            assume_role_arn arn:aws:iam::XXXXXXXXXXXX:role/XXXXXXXXXXXX
            assume_role_session_name fluentd-aggregator
          </endpoint>
          logstash_format true
          logstash_prefix sag-${record['ACCOUNT_ALIAS']}-${record['REGION'].gsub(' ', '-')}-${record['CLUSTER_NAME']}
          time_key @timestamp # dynamic field that will be created at log ingestion. Can be compared with "date" field which is log created time.
          request_timeout 5s
          resurrect_after 60s
          reconnect_on_error true # reset the connection on any error not only host unreachable. As ES is managed(behind proxy) this is imporntant
          reload_on_failure true # reload node addresses on failure for dead node scan
          reload_connections false # fluentd will try to reload host lists from elasticsearch, but as cluster is behind proxy it will fail
          sniffer_class_name "#{ENV['FLUENT_SNIFFER_CLASS_NAME'] || 'Fluent::Plugin::OpenSearchSimpleSniffer'}"
          with_transporter_log true
          emit_error_for_missing_id false
          include_index_in_url false
          prefer_oj_serializer false
          log_os_400_reason true
          bulk_message_request_threshold 20M
          templates {"cls": "/etc/fluent/config.d/05_cls-index-template.json"}
          template_overwrite true
          <buffer>
            @type file
            path /var/log/fluent/kubernetes-aws-elasticsearch.system.buffer
            flush_mode interval
            flush_interval 5s
            flush_at_shutdown true
            flush_thread_count 8
            flush_thread_interval 1.0
            flush_thread_burst_interval 1.0
            delayed_commit_timeout 60
            slow_flush_log_threshold 20.0
            overflow_action throw_exception
            retry_type exponential_backoff
            retry_forever false
            retry_timeout 72h
            retry_max_interval 30
            retry_randomize true
            chunk_limit_size 10MB
            queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT'] || '32'}"
            total_limit_size 28672MB
            chunk_full_threshold 0.95
            queued_chunks_limit_size 8
          </buffer>
        </store>
        <store>
          @type prometheus
          <metric>
            name fluentd_output_status_num_records_total
            type counter
            desc The total number of outgoing records
            <labels>
              tag ${tag}
              hostname ${hostname}
              REGION ${REGION}
              ACCOUNT_ALIAS ${ACCOUNT_ALIAS}
              CLUSTER_NAME ${CLUSTER_NAME}
            </labels>
          </metric>
        </store>
      </match>
    </label>
    <label @ERROR>
      <match **>
        @type stdout
      </match>
    </label>
  05_cls-index-template.json: |-
    {
      "index_patterns": [
        "sag-*"
      ],
      "order": 100,
      "settings": {
        "number_of_replicas": 1,
        "number_of_shards": 2
      }
    }
Opensearch access policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:XXXXXXXXXXXX:domain/XXXXXXXXXXXX/*"
    }
  ]
}

Expected Behavior or What you need to ask

Fluentd to authenticate with the given IAM role and communicate with Opensearch

Using Fluentd and OpenSearch plugin versions

  • OS version
    Amazon Linux 2
  • Bare Metal or within Docker or Kubernetes or others?
    Kubernetes
  • Fluentd v1.0 or later
    • paste result of fluentd --version or td-agent --version
kubectl exec -n cls fluentd-aggregator-0 -- fluentd --version
fluentd 1.16.3
  • OpenSearch plugin version
    • paste boot log of fluentd or td-agent
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluentd' version '1.16.3'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-concat' version '2.5.0'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-grok-parser' version '2.6.2'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.4.0'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-opensearch' version '1.1.5'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-parser-cri' version '0.1.1'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-prometheus' version '2.0.3'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-record-modifier' version '2.1.1'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-10-18 08:33:50 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-systemd' version '1.0.5'
2024-10-18 08:33:51 +0000 [debug]: fluent/log.rb:341:debug: adding store type="opensearch"
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'logstash_prefix sag-${record['ACCOUNT_ALIAS']}-${record['REGION'].gsub(' ', '-')}-${record['CLUSTER_NAME']}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: sag-${record['ACCOUNT_ALIAS']}-${record['REGION'].gsub(' ', '-')}-${record['CLUSTER_NAME']}' doesn't have tag placeholder
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] Need substitution:
2024-10-18 08:33:51 +0000 [debug]: [aws-opensearch-service] 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
Error retrieving instance profile credentials: Aws::InstanceProfileCredentials::Non200Response
/fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:119:in `rescue in initialize': unable to sign request without credentials set (Aws::Errors::MissingCredentialsError)
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:108:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:33:in `new'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:33:in `signer_for'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:45:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:27:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/user_agent.rb:69:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/retry_errors.rb:365:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/user_agent.rb:60:in `metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/retry_errors.rb:385:in `with_metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/retry_errors.rb:365:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/http_checksum.rb:20:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:137:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/request_compression.rb:94:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/request_compression.rb:104:in `with_metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/request_compression.rb:94:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/query/handler.rb:30:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/plugins/endpoints.rb:43:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/user_agent.rb:60:in `metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/plugins/endpoints.rb:57:in `with_metrics'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/plugins/endpoints.rb:43:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/endpoint.rb:46:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/request.rb:72:in `send_request'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/client.rb:937:in `assume_role'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/assume_role_credentials.rb:65:in `refresh'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/refreshing_credentials.rb:28:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/assume_role_credentials.rb:53:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/out_opensearch.rb:238:in `new'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/out_opensearch.rb:238:in `aws_credentials'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/out_opensearch.rb:345:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:187:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/multi_output.rb:99:in `each'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/multi_output.rb:99:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:187:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:132:in `add_match'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
	from /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'
/fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sigv4-1.10.0/lib/aws-sigv4/signer.rb:694:in `extract_credentials_provider': missing credentials, provide credentials with one of the following options: (Aws::Sigv4::Errors::MissingCredentialsError)
  - :access_key_id and :secret_access_key
  - :credentials
  - :credentials_provider
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sigv4-1.10.0/lib/aws-sigv4/signer.rb:138:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:109:in `new'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:109:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:33:in `new'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:33:in `signer_for'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/sign.rb:45:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:27:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/user_agent.rb:69:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/retry_errors.rb:365:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/user_agent.rb:60:in `metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/retry_errors.rb:385:in `with_metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/retry_errors.rb:365:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/http_checksum.rb:20:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:137:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/request_compression.rb:94:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/request_compression.rb:104:in `with_metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/request_compression.rb:94:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/query/handler.rb:30:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/plugins/endpoints.rb:43:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/user_agent.rb:60:in `metric'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/plugins/endpoints.rb:57:in `with_metrics'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/plugins/endpoints.rb:43:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/endpoint.rb:46:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/seahorse/client/request.rb:72:in `send_request'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-sts/client.rb:937:in `assume_role'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/assume_role_credentials.rb:65:in `refresh'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/refreshing_credentials.rb:28:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.209.1/lib/aws-sdk-core/assume_role_credentials.rb:53:in `initialize'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/out_opensearch.rb:238:in `new'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/out_opensearch.rb:238:in `aws_credentials'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/out_opensearch.rb:345:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:187:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/multi_output.rb:99:in `each'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/multi_output.rb:99:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:187:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:132:in `add_match'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
	from /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.8.7)
      public_suffix (>= 2.0.2, < 7.0)
    aws-eventstream (1.3.0)
    aws-partitions (1.991.0)
    aws-sdk-core (3.209.1)
      aws-eventstream (~> 1, >= 1.3.0)
      aws-partitions (~> 1, >= 1.651.0)
      aws-sigv4 (~> 1.9)
      jmespath (~> 1, >= 1.6.1)
    aws-sigv4 (1.10.0)
      aws-eventstream (~> 1, >= 1.0.2)
    base64 (0.2.0)
    concurrent-ruby (1.3.4)
    cool.io (1.9.0)
    domain_name (0.6.20240107)
    excon (0.112.0)
    faraday (2.8.1)
      base64
      faraday-net_http (>= 2.0, < 3.1)
      ruby2_keywords (>= 0.0.4)
    faraday-excon (2.1.0)
      excon (>= 0.27.4)
      faraday (~> 2.0)
    faraday-net_http (3.0.2)
    faraday_middleware-aws-sigv4 (1.0.1)
      aws-sigv4 (~> 1.0)
      faraday (>= 2.0, < 3)
    ffi (1.17.0)
    ffi-compiler (1.3.2)
      ffi (>= 1.15.5)
      rake
    fluent-config-regexp-type (1.0.0)
      fluentd (> 1.0.0, < 2)
    fluent-plugin-concat (2.5.0)
      fluentd (>= 0.14.0, < 2)
    fluent-plugin-dedot_filter (1.0.0)
      fluentd (>= 0.14.0, < 2)
    fluent-plugin-detect-exceptions (0.0.15)
      fluentd (>= 0.10)
    fluent-plugin-grok-parser (2.6.2)
      fluentd (>= 0.14.6, < 2)
    fluent-plugin-json-in-json-2 (1.0.2)
      fluentd (>= 0.14.0, < 2)
      yajl-ruby (~> 1.0)
    fluent-plugin-kubernetes_metadata_filter (3.4.0)
      fluentd (>= 0.14.0, < 1.17)
      kubeclient (>= 4.0.0, < 5.0.0)
      lru_redux
    fluent-plugin-multi-format-parser (1.0.0)
      fluentd (>= 0.14.0, < 2)
    fluent-plugin-opensearch (1.1.5)
      aws-sdk-core (~> 3)
      excon
      faraday (>= 2.0.0)
      faraday-excon (>= 2.0.0)
      faraday_middleware-aws-sigv4 (~> 1.0.1)
      fluentd (>= 0.14.22)
      opensearch-ruby (>= 3.0.1)
    fluent-plugin-parser-cri (0.1.1)
      fluentd (>= 1)
    fluent-plugin-prometheus (2.0.3)
      fluentd (>= 1.9.1, < 2)
      prometheus-client (>= 2.1.0)
    fluent-plugin-record-modifier (2.1.1)
      fluentd (>= 1.0, < 2)
    fluent-plugin-rewrite-tag-filter (2.4.0)
      fluent-config-regexp-type
      fluentd (>= 0.14.2, < 2)
    fluent-plugin-systemd (1.0.5)
      fluentd (>= 0.14.11, < 2)
      systemd-journal (~> 1.4.2)
    fluentd (1.16.3)
      bundler
      cool.io (>= 1.4.5, < 2.0.0)
      http_parser.rb (>= 0.5.1, < 0.9.0)
      msgpack (>= 1.3.1, < 2.0.0)
      serverengine (>= 2.3.2, < 3.0.0)
      sigdump (~> 0.2.5)
      strptime (>= 0.2.4, < 1.0.0)
      tzinfo (>= 1.0, < 3.0)
      tzinfo-data (~> 1.0)
      webrick (~> 1.4)
      yajl-ruby (~> 1.0)
    http (4.4.1)
      addressable (~> 2.3)
      http-cookie (~> 1.0)
      http-form_data (~> 2.2)
      http-parser (~> 1.2.0)
    http-accept (1.7.0)
    http-cookie (1.0.7)
      domain_name (~> 0.5)
    http-form_data (2.3.0)
    http-parser (1.2.3)
      ffi-compiler (>= 1.0, < 2.0)
    http_parser.rb (0.8.0)
    jmespath (1.6.2)
    jsonpath (1.1.5)
      multi_json
    kubeclient (4.9.3)
      http (>= 3.0, < 5.0)
      jsonpath (~> 1.0)
      recursive-open-struct (~> 1.1, >= 1.1.1)
      rest-client (~> 2.0)
    logger (1.6.1)
    lru_redux (1.1.0)
    mime-types (3.6.0)
      logger
      mime-types-data (~> 3.2015)
    mime-types-data (3.2024.1001)
    msgpack (1.7.3)
    multi_json (1.15.0)
    netrc (0.11.0)
    oj (3.11.0)
    opensearch-ruby (3.4.0)
      faraday (>= 1.0, < 3)
      multi_json (>= 1.0)
    ostruct (0.6.0)
    prometheus-client (4.2.3)
      base64
    public_suffix (5.1.1)
    rake (13.2.1)
    recursive-open-struct (1.3.1)
      ostruct
    rest-client (2.1.0)
      http-accept (>= 1.7.0, < 2.0)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    rexml (3.3.8)
    ruby2_keywords (0.0.5)
    serverengine (2.3.2)
      sigdump (~> 0.2.2)
    sigdump (0.2.5)
    strptime (0.2.5)
    systemd-journal (1.4.2)
      ffi (~> 1.9)
    tzinfo (2.0.6)
      concurrent-ruby (~> 1.0)
    tzinfo-data (1.2024.2)
      tzinfo (>= 1.0.0)
    webrick (1.8.2)
    yajl-ruby (1.4.3)

PLATFORMS
  x86_64-linux

DEPENDENCIES
  ffi
  fluent-plugin-concat (~> 2.5.0)
  fluent-plugin-dedot_filter (~> 1.0)
  fluent-plugin-detect-exceptions (~> 0.0.13)
  fluent-plugin-grok-parser (~> 2.6.2)
  fluent-plugin-json-in-json-2 (>= 1.0.2)
  fluent-plugin-kubernetes_metadata_filter (~> 3.4.0)
  fluent-plugin-multi-format-parser (~> 1.0.0)
  fluent-plugin-opensearch (~> 1.1, >= 1.1.5)
  fluent-plugin-parser-cri (~> 0.1.0)
  fluent-plugin-prometheus (~> 2.0.2)
  fluent-plugin-record-modifier (~> 2.1.0)
  fluent-plugin-rewrite-tag-filter (~> 2.4.0)
  fluent-plugin-systemd (~> 1.0.5)
  fluentd (= 1.16.3)
  kubeclient (~> 4.9.3)
  oj (= 3.11.0)
  rexml (~> 3.3, >= 3.3.6)

BUNDLED WITH
   2.4.17
  • OpenSearch version (optional)
    2.13
  • OpenSearch template(s) (optional)
@DesislavaNPetrova
Copy link

Any updates here?

@vdurmishev
Copy link

Please advise if there are any updates here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@serapch @DesislavaNPetrova @vdurmishev