[apm] Update APM feature roles docs

[colleenmcginnis]

In #3980 (comment), @endorama suggested that we clarify the docs on APM feature roles. Let me know what you think of this approach. I attempted to clarify how feature roles work and which privileges are required for different use cases by:

• Reframing the intro to feature roles to emphasize that the what we're recommending is to create several feature-related roles and then assign one or more of these roles to each user (or group) based on which features the user needs to access.
• Providing an example of when a user might need to be assigned multiple roles from this list of feature roles (this needs work).
• Emphasizing that the Central configuration management role is required in most cases (unless central configuration management is explicitly disabled by the org/user).
• Putting all roles on a single page to increase the likelihood that the reader will see the context provided in the intro paragraphs.
• Adding a step at the end of the instructions for creating each role to remind users that they can assign other APM feature roles to the users as needed.

Here's what hasn't been addressed yet:

• Verify that the privileges listed in each role are correct and reflect the latest version of APM / Elastic as a whole. (I'm not sure how to go about confirming or testing these.)
• Are these the right feature roles? Are any missing? Should any be removed?

Here's the PR preview: https://observability-docs_bk_4193.docs-preview.app.elstc.co/guide/en/observability/master/apm-feature-roles.html

@endorama let me know what you think of this approach. If we're on the right track, we can iterate in this PR.

cc @simitt @carsonip

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[carsonip]

Verify that the privileges listed in each role are correct and reflect the latest version of APM / Elastic as a whole. (I'm not sure how to go about confirming or testing these.)
Are these the right feature roles? Are any missing? Should any be removed?

Our team should confirm the required privileges. IIUC points to update:

1. missing privilege to query cluster uuid
2. mark api key role as deprecated

[endorama]

Left some comment but overall is already a great improvement 🤩

[endorama]

I'll take on me to:

• Verify that the privileges listed in each role are correct and reflect the latest version of APM / Elastic as a whole. (I'm not sure how to go about confirming or testing these.)
• Are these the right feature roles? Are any missing? Should any be removed?
• confirm the required privileges to query cluster uuid.
• confirm if API key role should be mark as deprecated (and if yes when deprecation happened)

[endorama]

TL;DR: querying cluster uuid requires monitor permission if Elasticsearch security features are enabled or none if not.

Today I researched how to query cluster UUID. My understanding is that apm-server queries Elasticsearch on the / path to obtain it. From the documentation

f the Elasticsearch security features are enabled, you must have the monitor, manage, or all cluster privilege to use this API.

[endorama]

Regarding the Api Key role, my understanding is that is only needed for the apikey command in apm-server. This command has never been labelled GA, is deprecated since 8.6.0 and will be removed. For this command to work priviliges manage_own_api_key are required. The users should leverage Kibana and Elasticsearch APIs instead.

[endorama]

I've completed checking for roles/permissions.

• 🟩 source map role correctly covers using APM source maps in RUM
• 🟩 privileges checks are allowed without authentication in APM use case (checking own permissions)
• 🟨 cluster UUID checks are always performed as part of APM server startup preconditions; this check may require monitor permissions if Elasticsearch Security is enabled
• 🟨 if Tail Based Sampling is enabled a license check is performed; this check requires monitor permissions
• 🟩 writer role correctly covers ingesting data from APM Server to Elasticsearch
• 🟥 central config role should correctly cover using Central Config, but I was not able to test it. The current docs mention setting allow_restricted_indices to true but I was not able to understand how to do that from the Roles UI when creating a role. @colleenmcginnis if you have any clarification/input here I'd like to test this one too.

Details

• SEARCH .apm-source-map here: covered by source map role
  • used here
• HasPrivileges request here: does not require privileges
  • HasPrivileges API
    • All users can use this API, but only to determine their own privileges.
      To check the privileges of other users, you must use the run as feature.
• GET / here:
  • check is performed as part of every run when checking preconditions here
• GET _license here
  • only requested if tail sampling is enabled here
  • Get license APIrequires monitor permissions
• Bulk here (in go-docappender):
  • Bulk is performed as part of a docappender
    • BulkIndexerItems are added to the configured appedender here
• SEARCH .apm-agent-configuration here: covered by central config role but I wasn't able to test it because in the UI there seems to be no way to set allow_restricted_indices=true

[colleenmcginnis]

The current docs mention setting allow_restricted_indices to true but I was not able to understand how to do that from the Roles UI when creating a role. @colleenmcginnis if you have any clarification/input here I'd like to test this one too.

It looks like that setting was added about 7 months ago by @carsonip in 4834fce. Maybe Carson can provide context? (The related PR and issue don't provide much information.)

[colleenmcginnis]

Also @endorama ...

cluster UUID checks are always performed as part of APM server startup preconditions; this check may require monitor permissions if Elasticsearch Security is enabled

... can you confirm if this just for the writer role or if it is necessary for all roles? The phrase APM server startup preconditions makes me think it's necessary to do anything, but I wanted to make sure. Also is monitor a cluster or index privileges? It looks like there are both in https://elastic.co/guide/en/elasticsearch/reference/master/security-privileges.html.

[endorama]

@colleenmcginnis Sorry for the confusion! monitor refers to the cluster privilege. Being it a cluster privilege it's not necessary on all roles but at least 1 needs to have it if:

• security is enabled
• tail based sampling is used

I think adding it to the writer role make sense as the checks are performed as part of the usual APM server startup process.

[carsonip]

The current docs mention setting allow_restricted_indices to true but I was not able to understand how to do that from the Roles UI when creating a role

Apparently there is no option to set allow_restricted_indices in roles UI. Roles created in this UI will have allow_restricted_indices default to false. e.g.

{
  "agentcfg_1": {
    "cluster": [],
    "indices": [
      {
        "names": [
          ".apm-agent-configuration"
        ],
        "privileges": [
          "read"
        ],
        "field_security": {
          "grant": [
            "*"
          ],
          "except": []
        },
        "allow_restricted_indices": false
      }
    ],
    "applications": [],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}

However, if you create the API key from API Key UI, under "Control security privileges", the example config in the textbox will have allow_restricted_indices configurable. The last time I tested this, agentcfg requires allow_restricted_indices to be true. @endorama would be good if you confirm that again.

[endorama]

@carsonip I can confirm it's broken with allow_restricted_indices: false. I'll test it with the API key as described.

[endorama]

@carsonip Tested and it's working! While testing I also went down a rabbit hole for the API Key format that I documented in #4312 so we can address it separately.

[endorama]

🤩 looks great! Thank you!

[mdbirnstiehl]

Some little things and some questions about heading hierarchy.

[colleenmcginnis]

Some little things and some questions about heading hierarchy.

Thank you! I messed up the headings when I caught up to main today. 🙈

[mdbirnstiehl]

🦭 👍