Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hosts/auto-detection + K8s quick start plus overview #4073

Merged
merged 16 commits into from
Aug 13, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
96 changes: 96 additions & 0 deletions docs/en/serverless/quickstarts/auto-detect-logs-metrics.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
slug: /serverless/observability/quickstarts/auto-detect-logs-metrics
title: Auto-detect logs and metrics
description: Learn how to scan your hosts to detect and collect logs and metrics.
tags: [ 'serverless', 'observability', 'how-to' ]
---

<p><DocBadge template="technical preview" /></p>

In this quickstart guide, you'll learn how to scan your host to detect and collect logs and metrics,
then navigate to dashboards to further analyze and explore your observability data.
You'll also learn how to get value out of your observability data.

## Prerequisites and limitations

This quickstart has the following requirements and limitations:

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
- Root privileges on the host are required to run the auto-detection script used in this quickstart.
- The auto-detection script currently scans for metrics and logs from Apache, Docker, Nginx, and the host system.
It also scans for custom log files.
- The auto-detection script works on Linux and MacOS only and requires support for the `lsof` command.
dedemorton marked this conversation as resolved.
Show resolved Hide resolved
dedemorton marked this conversation as resolved.
Show resolved Hide resolved

## Detect and collect observability data

1. <DocLink slug="/serverless/observability/create-an-observability-project">Create a new ((observability)) project</DocLink>, or open an existing one.
1. In your ((observability)) project, go to **Add Data**.
1. Select **Collect and analyze logs**, and then select **Auto-detect logs and metrics**.
1. Copy the command that's shown. For example:
![Quick start showing command for running auto-detection](../images/quickstart-autodetection-command.png)
You'll run this command to download the auto-detection script and scan your system for observability data.
1. Open a terminal on the host you want to scan, and run the command.
1. Review the list of log files:
- Enter `Y` to ingest all the log files listed
- Enter `n` to either exclude log files or specify additional log paths. Enter `Y` to confirm your selections.

The script downloads and installs the software needed to collect observability data from the host and send it to Elastic.
dedemorton marked this conversation as resolved.
Show resolved Hide resolved
dedemorton marked this conversation as resolved.
Show resolved Hide resolved
When the script is done, you'll see a message like "((agent)) is configured and running."

There might be a slight delay before logs and other data are ingested.

## Visualize your data

After installation is complete and all relevant data is flowing into Elastic,
the **Visualize your data** section will show links to assets you can use to analyze your data.
Depending on what type of observability data was collected,
the page may show assets for the following integrations:

{/* TODO: Add more detail to this section to highlight " the salient points of the specific dashboards or pages." Waiting for info from SMEs. */}

<DocTable columns={[{ title: "Integration" }, { title: "What assets are available?" }]}>
<DocRow>
<DocCell>**System**</DocCell>
<DocCell>ADD DESCRIPTION HERE</DocCell>
</DocRow>
<DocRow>
<DocCell>**Apache**</DocCell>
<DocCell>ADD DESCRIPTION HERE</DocCell>
</DocRow>
<DocRow>
<DocCell>**Docker**</DocCell>
<DocCell>ADD DESCRIPTION HERE</DocCell>
</DocRow>
<DocRow>
<DocCell>**Nginx**</DocCell>
<DocCell>ADD DESCRIPTION HERE</DocCell>
</DocRow>
<DocRow>
<DocCell>**Custom .log files**</DocCell>
<DocCell>ADD DESCRIPTION HERE</DocCell>
</DocRow>
</DocTable>

## Get value out of your data

After using the dashboards to examine your data and confirm you've ingested all the host logs and metrics you want to monitor,
you can use Elastic ((observability)) to gain deeper insight into your data.

For host monitoring, the following capabilities and features are recommended:

- In the <DocLink slug="/serverless/observability/infrastructure-monitoring">Infrastructure UI</DocLink>, analyze and compare data collected from your hosts.
You can also:
- <DocLink slug="/serverless/observability/detect-metric-anomalies">Detect anomalies</DocLink> for memory usage and network traffic on hosts.
- <DocLink slug="/serverless/observability/alerting">Create alerts</DocLink> that notify you when an anomaly is detected or a metric exceeds a given value.
- In the <DocLink slug="/serverless/observability/discover-and-explore-logs">Logs Explorer</DocLink>, search and filter your log data,
get information about the structure of log fields, and display your findings in a visualization.
You can also:
- <DocLink slug="/serverless/observability/monitor-datasets">Monitor log data set quality</DocLink> to find degraded documents.
- <DocLink slug="/serverless/observability/run-log-pattern-analysis">Run a pattern analysis</DocLink> to find patterns in unstructured log messages.
- <DocLink slug="/serverless/observability/alerting">Create alerts</DocLink> that notify you when an Observability data type reaches or exceeds a given value.
- Use <DocLink slug="/serverless/observability/aiops">AIOps features</DocLink> to apply predictive analytics and machine learning to your data:
- <DocLink slug="/serverless/observability/aiops-detect-anomalies">Detect anomalies</DocLink> by comparing real-time and historical data from different sources to look for unusual, problematic patterns.
- <DocLink slug="/serverless/observability/aiops-analyze-spikes">Analyze log spikes and drops</DocLink>.
- <DocLink slug="/serverless/observability/aiops-detect-change-points">Detect change points</DocLink> in your time series data.

Refer to <DocLink slug="/serverless/observability/serverless-observability-overview"/> for a description of other useful features.
45 changes: 45 additions & 0 deletions docs/en/serverless/quickstarts/k8s-logs-metrics.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
slug: /serverless/observability/quickstarts/k8s-logs-metrics
title: Kubernetes logs and metrics
dedemorton marked this conversation as resolved.
Show resolved Hide resolved
description: Learn how to monitor your cluster infrastructure running on Kubernetes.
tags: [ 'serverless', 'observability', 'how-to' ]
---

<p><DocBadge template="technical preview" /></p>

In this quickstart guide, you'll learn how to create the Kubernetes resources that are required to monitor your cluster infrastructure.

This new approach requires minimal configuration and provides you with an easy setup to monitor your cluster. You no longer need to download, install, or configure the Elastic Agent, everything happens automatically when you run the kubectl command.

The kubectl command installs the Elastic Agent in your Kubernetes cluster, downloads all the Kubernetes resources needed to collect metrics from the cluster, and sends it to Elastic.

## Before you begin

This quickstart has the following requirements:

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to <DocLink slug="/serverless/general/assign-user-roles" />.
- You have a running Kubernetes cluster.
- You have installed Kubectl.
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

## Collect your data

1. <DocLink slug="/serverless/observability/create-an-observability-project">Create a new ((observability)) project</DocLink>, or open an existing one.
1. In your ((observability)) project, go to **Add Data**.
1. Select **Monitor infrastructure**, and then select **Kubernetes**.
1. Copy the command that's shown. For example:
![Quickstart showing command for kubernetes](../images/quickstart-k8s-command.png)
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved
1. Open a terminal on the cluster you want to monitor, and run the command.

The kubectl command generates an output similar to the following one:

![Kubernetes command output](../images/quickstart-k8s-command-output.png)
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

1. Go back to the **Add Observability Data** page, where you will see the message **We are monitoring your cluster**.
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

1. Click **Explore Kubernetes cluster** to navigate to dashboards and explore your data.

## Get value out of your data

TBD

Refer to <DocLink slug="/serverless/observability/serverless-observability-overview"/> for a description of other useful features.
20 changes: 20 additions & 0 deletions docs/en/serverless/quickstarts/overview.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
slug: /serverless/observability/quickstarts/overview
title: Quickstarts
description: Learn how to ingest your observability data and get immediate value.
tags: [ 'serverless', 'observability', 'how-to' ]
---

Our quickstarts dramatically reduce your time-to-value by offering a fast path to ingest and visualize your Observability data.
Each quickstart provides:

- A highly opinionated, fast path to data ingestion
- Sensible configuration defaults with minimal configuration required
- Auto-detection of logs and metrics
dedemorton marked this conversation as resolved.
Show resolved Hide resolved
- Quick access to related dashboards and visualizations

## Available quickstarts

- <DocLink slug="/serverless/observability/quickstarts/auto-detect-logs-metrics"/>
- <DocLink slug="/serverless/observability/quickstarts/k8s-logs-metrics"/>

11 changes: 11 additions & 0 deletions docs/en/serverless/serverless-observability.docnav.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@
"classic-sources": [ "enObservabilityObservabilityIntroduction" ],
"classic-skip": true
},
{
"slug": "/serverless/observability/quickstarts/overview",
"items": [
{
"slug": "/serverless/observability/quickstarts/auto-detect-logs-metrics"
},
{
"slug": "/serverless/observability/quickstarts/k8s-logs-metrics"
}
]
},
{
"slug": "/serverless/observability/observability-billing"
},
Expand Down
5 changes: 5 additions & 0 deletions docs/en/serverless/what-is-observability-serverless.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ layout: landing
sectionTitle="Get started"
items={
[
{
"title": "Quickstarts",
slug: "/serverless/observability/quickstarts/overview",
"description": "Learn how to ingest your observability data and get immediate value.",
},
{
"title": "Get started with Logs",
slug: "/serverless/observability/get-started-with-logs",
Expand Down
Loading