Skip to content

Create central "Elastic Stack settings" page #1179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Starting in {{stack}} version 8.0, how you change APM settings and the settings
Standalone APM Server (legacy)
: Deployments created prior to {{stack}} version 8.0 are in legacy mode. Upgrading to or past {{stack}} 8.0 does not remove you from legacy mode.

Check [Edit standalone APM settings (legacy)](#ece-edit-apm-standalone-settings-ece)for information on how to configure Elastic APM in this mode.
Check [Edit standalone APM settings (legacy)](#ece-edit-apm-standalone-settings-ece) for information on how to configure Elastic APM in this mode.


To learn more about the differences between these modes, or to switch from Standalone APM Server (legacy) mode to {{fleet}}-managed, check [Switch to the Elastic APM integration](/solutions/observability/apm/switch-to-elastic-apm-integration.md).
Expand Down
39 changes: 39 additions & 0 deletions deploy-manage/deploy/cloud-on-k8s/_snippets/es-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
Any setting defined in the `elasticsearch.yml` configuration file can also be defined for a set of {{es}} nodes in the `spec.nodeSets[?].config` section.

Some settings are managed by ECK. It is not recommended to change these managed settings. For a complete list, refer to [Settings managed by ECK](/deploy-manage/deploy/cloud-on-k8s/settings-managed-by-eck.md).

```yaml
spec:
nodeSets:
- name: masters
count: 3
config:
# On Elasticsearch versions before 7.9.0, replace the node.roles configuration with the following:
# node.master: true
node.roles: ["master"]
xpack.ml.enabled: true
- name: data
count: 10
config:
# On Elasticsearch versions before 7.9.0, replace the node.roles configuration with the following:
# node.master: false
# node.data: true
# node.ingest: true
# node.ml: true
# node.transform: true
node.roles: ["data", "ingest", "ml", "transform"]
```

:::{warning}
ECK parses {{es}} configuration and normalizes it to YAML. Consequently, some {{es}} configuration schema are impossible to express with ECK and, therefore, must be set using [dynamic cluster settings](/deploy-manage/deploy/self-managed/configure-elasticsearch.md#cluster-setting-types). For example:
```yaml
spec:
nodeSets:
- name: data
# ...
config:
cluster.max_shards_per_node: 1000
cluster.max_shards_per_node.frozen: 1000 # This won't work because cluster.max_shards_per_node is defined as a scalar value on the previous line
# ...
```
:::
18 changes: 18 additions & 0 deletions deploy-manage/deploy/cloud-on-k8s/_snippets/kib-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
You can add your own {{kib}} settings to the `spec.config` section.

The following example demonstrates how to set the [`elasticsearch.requestHeadersWhitelist`](kibana://reference/configuration-reference/general-settings.md#elasticsearch-requestheaderswhitelist) configuration option.

```yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-sample
spec:
version: 8.16.1
count: 1
elasticsearchRef:
name: "elasticsearch-sample"
config:
elasticsearch.requestHeadersWhitelist:
- authorization
```
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Starting from ECK `2.6.1` and {{es}} `8.6.1`, {{stack}} configuration policies a
* [Index templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template)
* [Components templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template)
* [Role mappings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping)
* [{{es}} Configuration](/deploy-manage/deploy/self-managed/configure-elasticsearch.md) (configuration settings for {{es}} that will go into `elasticsearch.yml`) [ECK 2.11.0]
* [{{es}} Configuration](/deploy-manage/deploy/self-managed/configure-elasticsearch.md) (configuration settings for {{es}} that will go into [`elasticsearch.yml`](/deploy-manage/stack-settings.md)) [ECK 2.11.0]
* [{{es}} Secure Settings](../../security/secure-settings.md) [ECK 2.11.0]
* [Secret Mounts](#k8s-stack-config-policy-specifics-secret-mounts) [ECK 2.11.0]

Expand Down Expand Up @@ -58,13 +58,13 @@ At least one of `spec.elasticsearch` or `spec.kibana` needs to be defined with a
* `indexLifecyclePolicies` are index lifecycle policies, to automatically manage the index lifecycle.
* `indexTemplates.componentTemplates` are component templates that are building blocks for constructing index templates that specify index mappings, settings, and aliases.
* `indexTemplates.composableIndexTemplates` are index templates to define settings, mappings, and aliases that can be applied automatically to new indices.
* `config` are the settings that go into the `elasticsearch.yml` file.
* `config` are the settings that go into the [`elasticsearch.yml`](/deploy-manage/stack-settings.md) file.
* `secretMounts` are the additional user created secrets that need to be mounted to the {{es}} Pods.
* `secureSettings` is a list of Secrets containing Secure Settings to inject into the keystore(s) of the {{es}} cluster(s) to which this policy applies, similar to the [{{es}} Secure Settings](../../security/secure-settings.md).

* `spec.kibana` describes the settings to configure for {{kib}}.

* `config` are the settings that go into the `kibana.yml` file.
* `config` are the settings that go into the [`kibana.yml`](/deploy-manage/stack-settings.md) file.
* `secureSettings` is a list of Secrets containing Secure Settings to inject into the keystore(s) of the {{kib}} instance(s) to which this policy applies, similar to the [{{kib}} Secure Settings](../../security/k8s-secure-settings.md).


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,24 +57,8 @@ Check [Set compute resources for {{kib}}, Elastic Maps Server, APM Server and Lo

## {{kib}} configuration [k8s-kibana-configuration]

You can add your own {{kib}} settings to the `spec.config` section.

The following example demonstrates how to set the [`elasticsearch.requestHeadersWhitelist`](kibana://reference/configuration-reference/general-settings.md#elasticsearch-requestheaderswhitelist) configuration option.

```yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-sample
spec:
version: 8.16.1
count: 1
elasticsearchRef:
name: "elasticsearch-sample"
config:
elasticsearch.requestHeadersWhitelist:
- authorization
```
:::{include} _snippets/kib-config.md
:::


## Scale out a {{kib}} deployment [k8s-kibana-scaling]
Expand Down
44 changes: 3 additions & 41 deletions deploy-manage/deploy/cloud-on-k8s/node-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,45 +8,7 @@ mapped_pages:

# Node configuration [k8s-node-configuration]

Any setting defined in the `elasticsearch.yml` configuration file can also be defined for a set of {{es}} nodes in the `spec.nodeSets[?].config` section.

Some settings are managed by ECK, it is not recommended to change them, refer to [Settings managed by ECK](settings-managed-by-eck.md) for more details.

```yaml
spec:
nodeSets:
- name: masters
count: 3
config:
# On Elasticsearch versions before 7.9.0, replace the node.roles configuration with the following:
# node.master: true
node.roles: ["master"]
xpack.ml.enabled: true
- name: data
count: 10
config:
# On Elasticsearch versions before 7.9.0, replace the node.roles configuration with the following:
# node.master: false
# node.data: true
# node.ingest: true
# node.ml: true
# node.transform: true
node.roles: ["data", "ingest", "ml", "transform"]
```

::::{warning}
ECK parses {{es}} configuration and normalizes it to YAML. Consequently, some {{es}} configuration schema are impossible to express with ECK and, therefore, must be set using [dynamic cluster settings](/deploy-manage/deploy/self-managed/configure-elasticsearch.md#cluster-setting-types). For example:
```yaml
spec:
nodeSets:
- name: data
# ...
config:
cluster.max_shards_per_node: 1000
cluster.max_shards_per_node.frozen: 1000 # This won't work because cluster.max_shards_per_node is defined as a scalar value on the previous line
# ...
```
::::

For more information on {{es}} settings, check [Configuring Elasticsearch](/deploy-manage/deploy/self-managed/configure-elasticsearch.md).
:::{include} _snippets/es-config.md
:::

For more information on {{es}} settings, check [Configuring Elasticsearch](/deploy-manage/deploy/self-managed/configure-elasticsearch.md).
90 changes: 19 additions & 71 deletions deploy-manage/deploy/elastic-cloud/edit-stack-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,54 +17,29 @@ mapped_pages:

# Edit {{stack}} settings

% What needs to be done: Refine
From the {{ecloud}} Console you can customize {{es}}, {{kib}}, and related products to suit your needs. These editors append your changes to the appropriate YAML configuration file and they affect all users of that cluster.

% Use migrated content from existing pages that map to this page:
## Available settings

% - [ ] ./raw-migrated-files/cloud/cloud/ec-add-user-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud/ec-editing-user-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-editing-user-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud/ec-manage-apm-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-manage-apm-settings.md
% - [ ] ./raw-migrated-files/cloud/cloud/ec-manage-appsearch-settings.md
% Notes: specify cluster 8.x or lower
% - [ ] ./raw-migrated-files/cloud/cloud/ec-manage-enterprise-search-settings.md
% Notes: specify cluster 8.x or lower
### Elasticsearch settings

$$$ec-add-user-settings$$$

$$$ech-es-elasticsearch-settings$$$

$$$xpack-monitoring-history-duration$$$

$$$ech-edit-apm-standalone-settings$$$

$$$ech-apm-settings$$$

$$$csp-strict$$$

$$$ec-appsearch-settings$$$

$$$ec-es-elasticsearch-settings$$$
{{ech}} automatically rejects `elasticsearch.yml` settings that could break your cluster.

From the {{ecloud}} Console you can customize {{es}}, {{kib}}, and related products to suit your needs. These editors append your changes to the appropriate YAML configuration file and they affect all users of that cluster. In each editor you can:
For a list of supported settings, refer to the [{{es}} configuration reference](elasticsearch://reference/elasticsearch/configuration-reference/index.md). Settings supported on {{ech}} are indicated by an {{ecloud}} icon (![logo cloud](https://doc-icons.s3.us-east-2.amazonaws.com/logo_cloud.svg "Supported on {{ecloud}}")).

### Kibana settings

## Edit {{es}} user settings [ec-add-user-settings]
{{ech}} supports most of the standard {{kib}} settings.

Change how {{es}} runs by providing your own user settings. {{ech}} appends these settings to each node’s `elasticsearch.yml` configuration file.
Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions.

{{ech}} automatically rejects `elasticsearch.yml` settings that could break your cluster.
For a list of supported settings, check [{{kib}} settings](kibana://reference/cloud/elastic-cloud-kibana-settings.md).

For a list of supported settings, check [Supported {{es}} settings](elasticsearch://reference/elasticsearch/configuration-reference/index.md).
### APM settings

::::{warning}
You can also update [dynamic cluster settings](../../../deploy-manage/deploy/self-managed/configure-elasticsearch.md#dynamic-cluster-setting) using {{es}}'s [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). However, {{ech}} doesn’t reject unsafe setting changes made using this API. Use it with caution.
::::
Refer to [APM configuration reference](/solutions/observability/apm/configure-apm-server.md) for information on how to configure the {{fleet}}-managed APM integration.

## Edit settings [ec-add-user-settings]

To add or edit user settings:

Expand All @@ -73,42 +48,15 @@ To add or edit user settings:

On the **Hosted deployments** page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

3. From your deployment menu, go to the **Edit** page.
4. In the **Elasticsearch** section, select **Manage user settings and extensions**.
5. Update the user settings.
6. Select **Save changes**.
1. Under the deployment's name in the navigation menu, select **Edit**.
2. Look for the **Manage user settings and extensions** and **Edit user settings** links for each deployment, and select the one corresponding to the component you want to update, such as {{es}} or {{kib}}.
3. Apply the necessary settings in the **Users Settings** tab of the editor and select **Back** when finished.
4. Select **Save** to apply the changes to the deployment. Saving your changes initiates a configuration plan change that restarts the affected components for you.

::::{note}
In some cases, you may get a warning saying "User settings are different across {{es}} instances". To fix this issue, ensure that your user settings (including the comments sections and whitespaces) are identical across all {{es}} nodes (not only the data tiers, but also the Master, Machine Learning, and Coordinating nodes).
::::

## Edit {{kib}} user settings [ec-manage-kibana-settings]

{{ech}} supports most of the standard {{kib}} and X-Pack settings. Through a YAML editor in the console, you can append {{kib}} properties to the `kibana.yml` file. Your changes to the configuration file are read on startup.

Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions.

For a list of supported settings, check [{{kib}} settings](kibana://reference/cloud/elastic-cloud-kibana-settings.md).

To change {{kib}} settings:

1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body).
2. Find your deployment on the home page or on the **Hosted deployments** page, then select **Manage** to access its settings menus.

On the **Hosted deployments** page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

3. From your deployment menu, go to the **Edit** page.
4. In the **Kibana** section, select **Edit user settings**. For deployments with existing user settings, you may have to expand the **Edit kibana.yml** caret instead.
5. Update the user settings.
6. Select **Save changes**.

Saving your changes initiates a configuration plan change that restarts {{kib}} automatically for you.

::::{note}
If a setting is not supported by {{ech}}, you will get an error message when you try to save.
::::

## Edit APM user settings [ec-manage-apm-settings]

Change how Elastic APM runs by providing your own user settings.
Check [APM configuration reference](/solutions/observability/apm/configure-apm-server.md) for information on how to configure the {{fleet}}-managed APM integration.
::::{warning}
You can also update [dynamic cluster settings](../../../deploy-manage/deploy/self-managed/configure-elasticsearch.md#dynamic-cluster-setting) using {{es}}'s [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). However, {{ech}} doesn’t reject unsafe setting changes made using this API. Use it with caution.
::::
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
When you start {{es}} for the first time, it automatically performs the following security setup:

* Generates [TLS certificates](#stack-security-certificates) for the [transport and HTTP layers](/deploy-manage/security/secure-cluster-communications.md#communication-channels)
* Applies TLS configuration settings to `elasticsearch.yml`
* Applies TLS configuration settings to [`elasticsearch.yml`](/deploy-manage/stack-settings.md)
* Creates an enrollment token to securely connect {{kib}} to {{es}}

You can then start {{kib}} and enter the enrollment token, which is valid for 30 minutes. This token automatically applies the security settings from your {{es}} cluster, authenticates to {{es}} with the built-in `kibana` service account, and writes the security configuration to `kibana.yml`.
You can then start {{kib}} and enter the enrollment token, which is valid for 30 minutes. This token automatically applies the security settings from your {{es}} cluster, authenticates to {{es}} with the built-in `kibana` service account, and writes the security configuration to [`kibana.yml`](/deploy-manage/stack-settings.md).

::::{note}
There are [some cases](/deploy-manage/security/self-auto-setup.md#stack-skip-auto-configuration) where security can’t be configured automatically because the node startup process detects that the node is already part of a cluster, or that security is already configured or explicitly disabled.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
When you start your first {{es}} node for the first time, it automatically performs the following security setup:

* Generates [TLS certificates](#stack-security-certificates) for the [transport and HTTP layers](/deploy-manage/security/secure-cluster-communications.md#communication-channels)
* Applies TLS configuration settings to `elasticsearch.yml`
* Applies TLS configuration settings to [`elasticsearch.yml`](/deploy-manage/stack-settings.md)
* Sets a password for the `elastic` superuser
* Creates an enrollment token to securely connect {{kib}} to {{es}}

You can then start {{kib}} and enter the enrollment token, which is valid for 30 minutes. This token automatically applies the security settings from your {{es}} cluster, authenticates to {{es}} with the built-in `kibana` service account, and writes the security configuration to `kibana.yml`.
You can then start {{kib}} and enter the enrollment token, which is valid for 30 minutes. This token automatically applies the security settings from your {{es}} cluster, authenticates to {{es}} with the built-in `kibana` service account, and writes the security configuration to [`kibana.yml`](/deploy-manage/stack-settings.md).

::::{note}
There are [some cases](/deploy-manage/security/self-auto-setup.md#stack-skip-auto-configuration) where security can’t be configured automatically because the node startup process detects that the node is already part of a cluster, or that security is already configured or explicitly disabled.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Perform the following steps on each node in the cluster:

1. Open `elasticsearch.yml` in a text editor.
1. Open [`elasticsearch.yml`](/deploy-manage/deploy/self-managed/configure-elasticsearch.md) in a text editor.
2. Comment out or remove the `cluster.initial_master_nodes` setting, if present.
3. Update the `discovery.seed_hosts` value so it contains the IP address and port of each of the master-eligible {{es}} nodes in the cluster. In the first node in the cluster, you need to add the `discovery.seed_hosts` setting manually.
4. Optionally, [restart the {{es}} service](/deploy-manage/maintenance/start-stop-services/start-stop-elasticsearch.md) to validate your configuration changes.
Expand Down
29 changes: 29 additions & 0 deletions deploy-manage/deploy/self-managed/_snippets/config-file-format.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
The configuration format is [YAML](https://yaml.org/). Here is an example of changing the path of the data and logs directories in {{es}}:

```yaml
path:
data: /var/lib/elasticsearch
logs: /var/log/elasticsearch
```

Settings can also be flattened as follows:

```yaml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
```

In YAML, you can format non-scalar values as sequences:

```yaml
discovery.seed_hosts:
- 192.168.1.10:9300
- 192.168.1.11
- seeds.mydomain.com
```

Though less common, you can also format non-scalar values as arrays:

```yaml
discovery.seed_hosts: ["192.168.1.10:9300", "192.168.1.11", "seeds.mydomain.com"]
```
Loading
Loading