Skip to content

Add hosts and K8s quickstarts to stateful docs #4168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 7 additions & 0 deletions docs/en/observability/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ include::observability-introduction.asciidoc[leveloffset=+1]
// What's new
include::whats-new.asciidoc[leveloffset=+1]

// Quickstarts
include::quickstarts/quickstarts-overview.asciidoc[leveloffset=+1]

include::quickstarts/monitor-hosts-with-elastic-agent.asciidoc[leveloffset=+2]

include::quickstarts/monitor-k8s-logs-metrics.asciidoc[leveloffset=+2]

// Get started with Elastic Observability
include::observability-get-started.asciidoc[leveloffset=+1]

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
[[quickstart-monitor-hosts-with-elastic-agent]]
= Monitor hosts with {agent}

preview::[]

In this quickstart guide, you'll learn how to scan your host to detect and collect logs and metrics,
then navigate to dashboards to further analyze and explore your observability data.
You'll also learn how to get value out of your observability data.

To scan your host, you'll run an auto-detection script that downloads and installs {agent},
which is used to collect observability data from the host and send it to Elastic.

The script also generates an {agent} configuration file that you can use with your existing Infrastructure-as-Code tooling.

[discrete]
== Prerequisites

* A user with the `superuser` {ref}/built-in-roles.html[built-in role] or the privileges required to onboard data.
+
[%collapsible]
.Expand to view required privileges
====
* {ref}/security-privileges.html#privileges-list-cluster[**Cluster**]: `['monitor', 'manage_own_api_key']`
* {ref}/security-privileges.html#privileges-list-indices[**Index**]: `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }`
* {kibana-ref}/kibana-privileges.html[**Kibana**]: `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }`
====
* Root privileges on the host—required to run the auto-detection script used in this quickstart.

[discrete]
== Limitations

* The auto-detection script currently scans for metrics and logs from Apache, Docker, Nginx, and the host system.
It also scans for custom log files.
* The auto-detection script works on Linux and MacOS only. Support for the `lsof` command is also required if you want to detect custom log files.
* If you've installed Apache or Nginx in a non-standard location, you'll need to specify log file paths manually when you run the scan.
* Because Docker Desktop runs in a VM, its logs are not auto-detected.

[discrete]
== Collect your data

. In {kib}, go to **Observability** and click **Add Data**.
. Select **Collect and analyze logs**, and then select **Auto-detect logs and metrics**.
. Copy the command that's shown. For example:
+
[role="screenshot"]
image::images/quickstart-autodetection-command.png[Quick start showing command for running auto-detection]
+
You'll run this command to download the auto-detection script and scan your system for observability data.
. Open a terminal on the host you want to scan, and run the command.
. Review the list of log files:
* Enter `Y` to ingest all the log files listed.
* Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections.

When the script is done, you'll see a message like "{agent} is configured and running."

There might be a slight delay before logs and other data are ingested.

*****
**Need to scan your host again?**

You can re-run the script on the same host to detect additional logs.
The script will scan the host and reconfigure {agent} with any additional logs that are found.
If the script misses any custom logs, you can add them manually by entering `n` after the script has finished scanning the host.
*****

[discrete]
== Visualize your data

After installation is complete and all relevant data is flowing into Elastic,
the **Visualize your data** section will show links to assets you can use to analyze your data.
Depending on what type of observability data was collected,
the page may link to the following integration assets:

|====
| Integration asset | Description

| **System**
| Prebuilt dashboard for monitoring host status and health using system metrics.

| **Apache**
| Prebuilt dashboard for monitoring Apache HTTP server health using error and access log data.


| **Docker**
| Prebuilt dashboard for monitoring the status and health of Docker containers.


| **Nginx**
| Prebuilt dashboard for monitoring Nginx server health using error and access log data.


| **Custom .log files**
| Logs Explorer for analyzing custom logs.
|====

For example, you can navigate the **Host overview** dashboard to explore detailed metrics about system usage and throughput.
Metrics that indicate a possible problem are highlighted in red.

[role="screenshot"]
image::images/quickstart-host-overview.png[Host overview dashboard]

[discrete]
== Get value out of your data

After using the dashboards to examine your data and confirm you've ingested all the host logs and metrics you want to monitor,
you can use Elastic {observability} to gain deeper insight into your data.

For host monitoring, the following capabilities and features are recommended:

* In the <<analyze-metrics,Infrastructure UI>>, analyze and compare data collected from your hosts.
You can also:
** <<inspect-metric-anomalies,Detect anomalies>> for memory usage and network traffic on hosts.
** <<create-alerts,Create alerts>> that notify you when an anomaly is detected or a metric exceeds a given value.
* In the <<explore-logs,Logs Explorer>>, search and filter your log data,
get information about the structure of log fields, and display your findings in a visualization.
You can also:
** <<monitor-datasets,Monitor log data set quality>> to find degraded documents.
** {kibana-ref}/xpack-ml-aiops.html#log-pattern-analysis[Run a pattern analysis] to find patterns in unstructured log messages.
** <<create-alerts,Create alerts>> that notify you when an Observability data type reaches or exceeds a given value.
* Use {kibana-ref}/xpack-ml.html[machine learning] to apply predictive analytics to your data:
** {kibana-ref}/xpack-ml-anomalies.html[Detect anomalies] by comparing real-time and historical data from different sources to look for unusual, problematic patterns.
** {kibana-ref}/xpack-ml-aiops.html#log-rate-analysis[Analyze log spikes and drops].
** {kibana-ref}/xpack-ml-aiops.html#change-point-detection[Detect change points] in your time series data.

Refer to the <<observability-introduction>> for a description of other useful features.
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
[[monitor-k8s-logs-metrics-with-elastic-agent]]
= Monitor your Kubernetes cluster with {agent}

preview::[]

In this quickstart guide, you'll learn how to create the Kubernetes resources that are required to monitor your cluster infrastructure.

This new approach requires minimal configuration and provides you with an easy setup to monitor your infrastructure. You no longer need to download, install, or configure the Elastic Agent, everything happens automatically when you run the kubectl command.

The kubectl command installs the standalone Elastic Agent in your Kubernetes cluster, downloads all the Kubernetes resources needed to collect metrics from the cluster, and sends it to Elastic.

[discrete]
== Prerequisites

* A user with the `superuser` {ref}/built-in-roles.html[built-in role] or the privileges required to onboard data.
+
[%collapsible]
.Expand to view required privileges
====
* {ref}/security-privileges.html#privileges-list-cluster[**Cluster**]: `['monitor', 'manage_own_api_key']`
* {ref}/security-privileges.html#privileges-list-indices[**Index**]: `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }`
* {kibana-ref}/kibana-privileges.html[**Kibana**]: `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }`
====
* A running Kubernetes cluster.
* https://kubernetes.io/docs/reference/kubectl/[Kubectl].

[discrete]
== Collect your data

. In {kib}, go to **Observability** and click **Add Data**.

. Select **Monitor infrastructure**, and then select **Kubernetes**.
+
[role="screenshot"]
image::images/quickstart-k8s-entry-point.png[Kubernetes entry point]

. To install the Elastic Agent on your host, copy and run the install command.
+
You will use the kubectl command to download a manifest file, inject user's API key generated by {kib}, and create the Kubernetes resources.

. Go back to the **Add Observability Data** page.
+
There might be a slight delay before data is ingested. When ready, you will see the message **We are monitoring your cluster**.

. Click **Explore Kubernetes cluster** to navigate to dashboards and explore your data.

[discrete]
== Visualize your data

After installation is complete and all relevant data is flowing into Elastic,
the **Visualize your data** section allows you to access the Kubernetes Cluster Overview dashboard that can be used to monitor the health of the cluster.

[role="screenshot"]
image::images/quickstart-k8s-overview.png[Kubernetes overview dashboard]

Furthermore, you can access other useful prebuilt dashboards for monitoring Kubernetes resources, for example running pods per namespace, as well as the resources they consume, like CPU and memory.

Refer to <<observability-introduction>> for a description of other useful features.
16 changes: 16 additions & 0 deletions docs/en/observability/quickstarts/quickstarts-overview.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[[quickstarts-overview]]
= Quickstarts

Our quickstarts dramatically reduce your time-to-value by offering a fast path to ingest and visualize your Observability data.
Each quickstart provides:

* A highly opinionated, fast path to data ingestion
* Sensible configuration defaults with minimal configuration required
* Auto-detection of logs and metrics for monitoring hosts
* Quick access to related dashboards and visualizations

[discrete]
== Available quickstarts

* <<quickstart-monitor-hosts-with-elastic-agent>>
* <<monitor-k8s-logs-metrics-with-elastic-agent>>
10 changes: 4 additions & 6 deletions docs/en/serverless/quickstarts/k8s-logs-metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,11 @@ This new approach requires minimal configuration and provides you with an easy s

The kubectl command installs the standalone Elastic Agent in your Kubernetes cluster, downloads all the Kubernetes resources needed to collect metrics from the cluster, and sends it to Elastic.

## Before you begin
## Prerequisites

This quickstart has the following requirements:

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
- You have a running Kubernetes cluster.
- You have installed [Kubectl](https://kubernetes.io/docs/reference/kubectl/).
- A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
- A running Kubernetes cluster.
- [Kubectl](https://kubernetes.io/docs/reference/kubectl/).

## Collect your data

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,22 @@ You'll also learn how to get value out of your observability data.
To scan your host, you'll run an auto-detection script that downloads and installs ((agent)),
which is used to collect observability data from the host and send it to Elastic.

The script also generate an ((agent)) configuration file that you can use with your existing Infrastructure-as-Code tooling.
The script also generates an ((agent)) configuration file that you can use with your existing Infrastructure-as-Code tooling.

## Prerequisites and limitations
## Prerequisites

This quickstart has the following requirements and limitations:
- A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
- Root privileges on the host—required to run the auto-detection script used in this quickstart.

## Limitations

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
- Root privileges on the host are required to run the auto-detection script used in this quickstart.
- The auto-detection script currently scans for metrics and logs from Apache, Docker, Nginx, and the host system.
It also scans for custom log files.
- The auto-detection script works on Linux and MacOS only. Support for the `lsof` command is also required if you want to detect custom log files.
- If you've installed Apache or Nginx in a non-standard location, you'll need to specify log file paths manually when you run the scan.
- Because Docker Desktop runs in a VM, its logs are not auto-detected.

## Detect and collect observability data
## Collect your data

1. <DocLink slug="/serverless/observability/create-an-observability-project">Create a new ((observability)) project</DocLink>, or open an existing one.
1. In your ((observability)) project, go to **Add Data**.
Expand All @@ -38,7 +39,7 @@ This quickstart has the following requirements and limitations:
You'll run this command to download the auto-detection script and scan your system for observability data.
1. Open a terminal on the host you want to scan, and run the command.
1. Review the list of log files:
- Enter `Y` to ingest all the log files listed
- Enter `Y` to ingest all the log files listed.
- Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections.

When the script is done, you'll see a message like "((agent)) is configured and running."
Expand Down
Loading