Insufficient Permission to Create Index is a Debug Level Log

[Magnitus-]

(check apply)

• [ x ] read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

Fluentd Config:

<system>
  Log_Level debug
</system>

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

<match test>
  @type opensearch
  @log_level debug
  with_transporter_log true
  host 192.168.122.165
  port 9200
  scheme https
  ssl_verify true
  ssl_version TLSv1_2
  ca_file /opt/certs/opensearch-ca.pem
  client_cert /opt/certs/opensearch-fluentd.pem
  client_key /opt/certs/opensearch-fluentd.key
  index_name fluentd-test
</match>

Opensearch Permission: The "indices:admin/create" permission is missing from the opensearch index on the user's role, preventing the opensearch user fluentd uses from creating indices.

Message:

2022-10-16 15:09:48 +0000 [debug]: #0 < {"took":4,"errors":true,"items":[{"index":{"_index":"fluentd-test","_id":null,"status":403,"error":{"type":"security_exception","reason":"no permissions for [indices:admin/create] and User [name=fluentd, backend_roles=[], requestedTenant=null]"}}},

Expected Behavior or What you need to ask

I expected the error to appear even when logs are set to info.

Instead, everything appeared to be silently working (given fluentd's feedback), but the log indices were not getting created in opensearch.

The problem only became apparent once I set the logs to debug.

Using Fluentd and OpenSearch plugin versions

• OS version: Ubuntu Focal
• Bare Metal or within Docker or Kubernetes or others: Opensearch in kvm, fluentd in docker
• Fluentd v1.0 or later:
  • paste result of fluentd --version or td-agent --version: fluentd 1.14.0
• OpenSearch plugin version
  • paste boot log of fluentd or td-agent: gem 'fluent-plugin-opensearch' version '1.0.8'
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock:

*** LOCAL GEMS ***

async (1.30.1)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.9)
aws-eventstream (1.2.0)
aws-partitions (1.646.0)
aws-sdk-core (3.160.0)
aws-sigv4 (1.5.2)
bigdecimal (default: 1.4.1)
bundler (default: 1.17.2)
cmath (default: 1.0.0)
concurrent-ruby (1.1.10)
console (1.15.0)
cool.io (1.7.1)
csv (default: 3.0.9)
date (default: 2.0.3)
dbm (default: 1.0.0)
did_you_mean (1.3.0)
e2mmap (default: 0.1.0)
etc (default: 1.0.1)
excon (0.93.0)
ext_monitor (0.1.2)
faraday (1.10.2)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.1.0)
fluent-plugin-opensearch (1.0.8)
fluent-plugin-slack (0.6.7)
fluentd (1.14.0)
forwardable (default: 1.2.0)
gdbm (default: 2.0.0)
http_parser.rb (0.7.0)
io-console (default: 0.4.7)
ipaddr (default: 1.2.2)
irb (default: 1.0.0)
jmespath (1.6.1)
json (2.4.1, default: 2.1.0)
logger (default: 1.3.0)
matrix (default: 0.1.0)
minitest (5.11.3)
msgpack (1.5.1)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
oj (3.10.18)
opensearch-api (2.0.2)
opensearch-ruby (2.0.3)
opensearch-transport (2.0.1)
openssl (default: 2.1.2)
ostruct (default: 0.1.0)
power_assert (1.1.3)
prime (default: 0.1.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
psych (default: 3.1.0)
rake (12.3.3)
rdoc (default: 6.1.2.1)
rexml (default: 3.1.9.1)
rss (default: 0.2.7)
ruby2_keywords (0.0.5)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
serverengine (2.2.5)
shell (default: 0.7)
sigdump (0.2.4)
stringio (default: 0.0.2)
strptime (0.2.5)
strscan (default: 1.0.0)
sync (default: 0.5.0)
test-unit (3.2.9)
thwait (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
webrick (default: 1.4.4)
xmlrpc (0.3.0)
yajl-ruby (1.4.2)
zlib (default: 1.0.0)

• OpenSearch version (optional): 2.2.1
• OpenSearch template(s) (optional)

[cosmo0920]

This indicates that your OpenSearch cluster’s user does not have a permission for index creation:
Ref: https://opensearch.org/docs/latest/security-plugin/access-control/permissions/
Ref: https://opensearch.org/docs/latest/security-plugin/access-control/api#create-user

[Magnitus-]

This indicates that your OpenSearch cluster’s user does not have a permission for index creation: Ref: https://opensearch.org/docs/latest/security-plugin/access-control/permissions/ Ref: https://opensearch.org/docs/latest/security-plugin/access-control/api#create-user

The permission error was definitely a separate problem on our side that we have since resolved.

I think the main thing that seemed like a quirk of the provider to me in the incident was that I had to turn on debug level logs to seen the error message appear on the fluentd logs (as a debug log). The errors prior to that appeared with regular log levels set as error logs if memory serves.