Merge branch 'main' into monitor-datasets

.github/workflows/co-docs-builder.yml

@@ -20,12 +20,17 @@ on:
 
 jobs:
   publish:
-    if: github.event.label.name == 'ci:doc-build'
-    uses: elastic/workflows/.github/workflows/docs-elastic-co-publish.yml@main
+    if: contains(github.event.pull_request.labels.*.name, 'ci:doc-build')
+    uses: elastic/workflows/.github/workflows/docs-versioned-publish.yml@main
     with:
-      subdirectory: 'docs/en/serverless/'
+      # Refers to Vercel project
+      project-name: elastic-dot-co-docs-preview-docs
+      # Which prebuild step (dev or not)
+      prebuild: wordlake-docs
+      # Docsmobile project dir
+      site-repo: docs-site    
     secrets:
       VERCEL_GITHUB_TOKEN: ${{ secrets.VERCEL_GITHUB_TOKEN_PUBLIC }}
       VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN_PUBLIC }}
       VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID_PUBLIC }}
-      VERCEL_PROJECT_ID_DOCS_CO: ${{ secrets.VERCEL_PROJECT_ID_DOCS_CO_PUBLIC }}
+      VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID_ELASTIC_DOT_CO_DOCS_PRODUCTION_PUBLIC }}

.github/workflows/run-minor-release.yml

@@ -36,10 +36,8 @@ jobs:
           GH_TOKEN: ${{ github.token }}
 
       - if: ${{ failure() }}
-        uses: elastic/apm-pipeline-library/.github/actions/slack-message@current
+        uses: elastic/oblt-actions/slack/send@v1
         with:
-          url: ${{ secrets.VAULT_ADDR }}
-          roleId: ${{ secrets.VAULT_ROLE_ID }}
-          secretId: ${{ secrets.VAULT_SECRET_ID }}
-          channel: '#observablt-bots'
+          bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
+          channel-id: '#observablt-bots'
           message: ":traffic_cone: release automation failed for `${{ github.repository }}@${{ inputs.version }}`, @robots-ci please look what's going on <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|here>"

README.md

@@ -75,3 +75,16 @@ The Integrations Developer Guide is not versioned, and should never be backporte
 ## Reviews
 
 All documentation pull requests automatically add the **[@obs-docs](https://github.com/orgs/elastic/teams/obs-docs)** team as a reviewer.
+
+## License
+
+Shield: [![CC BY-NC-ND 4.0][cc-by-nc-nd-shield]][cc-by-nc-nd]
+
+This work is licensed under a
+[Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License][cc-by-nc-nd].
+
+[![CC BY-NC-ND 4.0][cc-by-nc-nd-image]][cc-by-nc-nd]
+
+[cc-by-nc-nd]: http://creativecommons.org/licenses/by-nc-nd/4.0/
+[cc-by-nc-nd-image]: https://licensebuttons.net/l/by-nc-nd/4.0/88x31.png
+[cc-by-nc-nd-shield]: https://img.shields.io/badge/License-CC%20BY--NC--ND%204.0-lightgrey.svg

docs/en/integrations/build-integration.asciidoc

@@ -400,18 +400,6 @@ The `event.module` and `event.dataset` fields are defined with a fixed value spe
 - `event.dataset: apache.access`
 Field `@timestamp` is defined here as type `date`.
 
-=== ecs.yml
-This file specifies every Elastic Common Schema (ECS) field used by the integration that is not defined in the files `agent.yml` or `base-fields.yml` files. It uses `external: ecs` references.
-For example:
-+
-[source,yaml]
-----
-- external: ecs
-  name: client.ip
-- external: ecs
-  name: destination.domain
-----
-
 === fields.yml
 Here we define fields that we need in our integration and are not found in the ECS.
 The example below defines field `apache.access.ssl.protocol` in the Apache integration.

docs/en/observability/apm/api-events.asciidoc

@@ -13,7 +13,7 @@ Events can be:
 * Metrics
 
 Each event is sent as its own line in the HTTP request body.
-This is known as http://ndjson.org[newline delimited JSON (NDJSON)].
+This is known as https://github.com/ndjson/ndjson-spec[newline delimited JSON (NDJSON)].
 
 With NDJSON, agents can open an HTTP POST request and use chunked encoding to stream events to the APM Server
 as soon as they are recorded in the agent.

docs/en/observability/cloud-monitoring/aws/monitor-amazon-intro.asciidoc

@@ -38,8 +38,12 @@ include::monitor-aws-vpc-flow-logs.asciidoc[leveloffset=+2]
 
 include::monitor-aws-cloudtrail-firehose.asciidoc[leveloffset=+2]
 
+include::monitor-aws-firewall-firehose.asciidoc[leveloffset=+2]
+
 include::monitor-aws-waf-firehose.asciidoc[leveloffset=+2]
 
+include::monitor-aws-cloudwatch-firehose.asciidoc[leveloffset=+2]
+
 include::monitor-aws-firehose-troubleshooting.asciidoc[leveloffset=+2]
 
 include::monitor-aws-esf.asciidoc[]

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

@@ -0,0 +1,204 @@
+[[monitor-aws-cloudwatch-firehose]]
+= Monitor CloudWatch logs
+
+++++
+<titleabbrev>Monitor CloudWatch logs</titleabbrev>
+++++
+
+In this section, you'll learn how to export log events from CloudWatch logs to an Elastic cluster by using Amazon Data Firehose.
+
+You'll go through the following steps:
+
+- Install AWS integration in {kib}
+- Select a CloudWatch log group to monitor 
+- Create a delivery stream in Amazon Data Firehose
+- Set up a subscription filter to forward the logs using the Firehose stream
+- Visualize your logs in {kib}
+
+[discrete]
+[[firehose-cloudwatch-prerequisites]]
+== Before you begin
+
+We assume that you already have:
+
+- An AWS account with permissions to pull the necessary data from AWS.
+- A deployment using our hosted {ess} on {ess-trial}[{ecloud}]. The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data. AWS Data Firehose works with Elastic Stack version 7.17 or greater, running on Elastic Cloud only.
+
+IMPORTANT: AWS PrivateLink is not supported. Make sure the deployment is on AWS, because the Amazon Data Firehose delivery stream connects specifically to an endpoint that needs to be on AWS.
+
+[discrete]
+[[firehose-cloudwatch-step-one]]
+== Step 1: Install AWS integration in {kib}
+
+. In {kib}, navigate to *Management* > *Integrations* and browse the catalog to find the AWS integration.
+
+. Navigate to the *Settings* tab and click *Install AWS assets*.
+
+[discrete]
+[[firehose-cloudwatch-step-two]]
+== Step 2: Select a CloudWatch log group to monitor 
+
+image::firehose-cloudwatch-log-group.png[CloudWatch log group]
+
+In this tutorial, you are going to collect application logs from an AWS Lambda-based app and forward them to Elastic. 
+
+**Create a Lambda function**
+
+NOTE: You can skip this section if you already have a Lambda function, or any other service or application that sends logs to a CloudWatch log group. Take note of the log group from which you want to collect log events and move to the next section. 
+
+Like many other services and platforms in AWS, Lambda functions natively log directly to CloudWatch out of the box. 
+
+. Go to the https://console.aws.amazon.com/[AWS console] and open the AWS Lambda page.
+. Click **Create function** and select the option to create a function from scratch.
+. Select a **Function name**.
+. As a **Runtime**, select a recent version of Python. For example, Python 3.11.
+. Select your **Architecture** of choice between `arm64` and `x86_64`.
+. Confirm and create the Lambda function.
++
+When AWS finishes creating the function, go to the **Code source** section and paste the following Python code as function source code:
++
+[source,python]
+----
+import json
+
+def lambda_handler(event, context):
+    print("Received event: " + json.dumps(event))
+----
+
+. Click **Deploy** to deploy the changes to the source code.
+
+**Generate some sample logs**
+
+With the function ready to go, you can invoke it a few times to generate sample logs.
+On the function page, follow these steps:
+
+. Select **Test**.
+. Select the option to create a new test event.
+. Name the test event and **Save** the changes.
+. Click the **Test** button to execute the function.
+
+Visit the function's log group. Usually, the AWS console offers a handy link to jump straight to the log group it created for this function's logs.
+You should get something similar to the following:
+
+image::firehose-cloudwatch-sample-logs.png[CloudWatch log group with sample logs]
+
+Take note of the log group name for this Lambda function, as you will need it in the next steps.
+
+[discrete]
+[[firehose-cloudwatch-step-three]]
+== Step 3: Create a stream in Amazon Data Firehose
+
+image::firehose-cloudwatch-firehose-stream.png[Amazon Firehose Stream]
+
+. Go to the https://console.aws.amazon.com/[AWS console] and navigate to Amazon Data Firehose.  
+
+. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`. 
+
+. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. 
++
+NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required.
+
+. In the **Destination settings** section, set the following parameter:
+`es_datastream_name` = `logs-aws.generic-default`
+
+The Firehose stream is now ready to send logs to your Elastic Cloud deployment.
+
+[discrete]
+[[firehose-cloudwatch-step-four]]
+== Step 4: Send Lambda function log events to a Firehose stream
+
+image::firehose-cloudwatch-subscription-filter.png[CloudWatch subscription filter]
+
+To send log events from CloudWatch to Firehose, open the log group where the Lambda service is logging and create a subscription filter.
+
+**Create a subscription filter for Amazon Data Firehose** 
+
+The https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html[subscription filter] allows you to pick log events from the log group and forward them to other services, such as an Amazon Kinesis stream, an Amazon Data Firehose stream, or AWS Lambda.
+
+. On the log group page, select *Subscription filters* and click the *Create Amazon Data Firehose subscription filter* button.
+
+From here, follow these steps:
+
+. Choose a destination. Select the Firehose stream you created in the previous step.
+
+. Grant the CloudWatch service permission to send log events to the stream in Firehose:
+
+.. Create a new role with a trust policy that allows CloudWatch service to assume the role.
+
+.. Assign a policy to the role that permits "putting records" into a Firehose  stream.
+
+. Create a new IAM role and use the following JSON as the trust policy:
++
+[source,json]
+----
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Principal": {
+                "Service": "logs.<REGION>.amazonaws.com"
+            },
+            "Action": "sts:AssumeRole",
+            "Condition": {
+                "StringLike": {
+                    "aws:SourceArn": "arn:aws:logs:<REGION>:<ACCOUNT_ID>:*"
+                }
+            }
+        }
+    ]
+}
+----
+
+. Assign a policy to the IAM role by using the following JSON file:
++
+[source,json]
+----
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": "firehose:PutRecord",
+            "Resource": "arn:aws:firehose:<REGION>:<ACCOUNT_ID>:deliverystream/<YOUR_FIREHOSE_STREAM>"
+        }
+    ]
+}
+----
+
+When the new role is ready, you can select it in the subscription filter.
+
+. Configure log format and filters. Select the "Other" in the **Log format** option.
+
+. Set log format and filters
++
+If you want to forward all log events, you can empty the filter pattern. You can use the *Subscription filter pattern* to forward only the log events that match the pattern. The *Test pattern* tool on the same page allows you to test filter patterns before creating the subscription filter.
+
+. Generate additional logs.
++
+Open the AWS Lambda page again, select the function you created, and execute it a few times to generate new log events.
+
+**Check if there are destination error logs**
+
+On the https://console.aws.amazon.com/[AWS console], navigate to your Firehose stream and check for entries in the *Destination error logs* section.
+
+If everything is running smoothly, this list is empty. If there's an error, you can check the details. The following example shows a delivery stream that fails to send records to the Elastic stack due to bad authentication settings:
+
+image::firehose-cloudwatch-destination-errors.png[Firehose destination errors]
+
+The Firehose delivery stream reports:
+
+* The number of failed deliveries. 
+* The failure detail. 
+
+[discrete]
+[[firehose-cloudwatch-step-five]]
+== Step 5: Visualize your logs in {kib}
+
+image::firehose-cloudwatch-data-stream.png[Vizualize logs in Kibana]
+
+With the logs streaming to the Elastic stack, you can now visualize them in {kib}.
+
+In {kib}, navigate to the *Discover* page and select the index pattern that matches the Firehose stream name. Here is a sample of logs from the Lambda function you forwarded to the `logs-aws.generic-default` data stream:
+
+image::firehose-cloudwatch-verify-discover.png[Sample logs in Discover]

docs/en/observability/cloud-monitoring/aws/monitor-aws-firehose.asciidoc

@@ -5,7 +5,7 @@
 <titleabbrev>Monitor {aws} with Amazon Data Firehose</titleabbrev>
 ++++
 
-Amazon Data Firehose is a popular service that allows you to send your VPC flow logs data to Elastic in minutes without a single line of code and without building or managing your own data ingestion and delivery infrastructure. Amazon Data Firehose Helps you answer questions like what percentage of your traffic is getting dropped, and how much traffic is getting generated for specific sources and destinations.
+Amazon Data Firehose is a popular service that allows you to send your service logs to Elastic in minutes without a single line of code and without building or managing your own data ingestion and delivery infrastructure.
 
 [discrete]
 [[aws-elastic-firehose-what-you-learn]]
@@ -22,8 +22,8 @@ In this tutorial, you'll learn how to:
 [[aws-elastic-firehose-before-you-begin]]
 === Before you begin
 
-Create a deployment using our hosted {ess} on {ess-trial}[{ecloud}].
-The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data. You also need an AWS account with permissions to pull the necessary data from AWS.
+Create a deployment in AWS regions (including gov cloud) using our hosted {ess} on {ess-trial}[{ecloud}].
+The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data.
 
 [discrete]
 [[firehose-step-one]]
@@ -33,9 +33,7 @@ The deployment includes an {es} cluster for storing and searching your data, and
 
 . Navigate to the *Settings* tab and click *Install AWS assets*. Confirm by clicking *Install AWS* in the popup.
 
-. Install AWS Firehose integration assets in Kibana. 
-
-NOTE: Firehose integration is currently in beta. Make sure to enable *Display beta integrations*.
+. Install Amazon Data Firehose integration assets in Kibana.
 
 [discrete]
 [[firehose-step-two]]
@@ -65,21 +63,23 @@ NOTE: For advanced use cases, source records can be transformed by invoking a cu
 +
 * *Parameters*:
 +
-  ** `es_datastream_name`: Elastic recommends setting the `es_datastream_name` parameter to `logs-awsfirehose-default` to leverage the routing rules defined in this integration. If this parameter is not specified, data is sent to the `logs-generic-default` data stream by default.
+  ** `es_datastream_name`: This parameter is optional and can be used to set which data stream documents will be stored. If this parameter is not specified, data is sent to the `logs-awsfirehose-default` data stream by default.
   ** `include_cw_extracted_fields`: This parameter is optional and can be set when using a CloudWatch logs subscription filter as the Firehose data source. When set to true, extracted fields generated by the filter pattern in the subscription filter will be collected. Setting this parameter can add many fields into each record and may significantly increase data volume in Elasticsearch. As such, use of this parameter should be carefully considered and used only when the extracted fields are required for specific filtering and/or aggregation.
-
+  ** `set_es_document_id`: This parameter is optional and can be set to allow Elasticsearch to assign each document a random ID or use a calculated unique ID for each document. Default is true. When set to false, a random ID will be used for each document which will help indexing performance.
 . In the *Backup settings* panel, it is recommended to configure S3 backup for failed records. It’s then possible to configure workflows to automatically retry failed records, for example by using {esf-ref}/aws-elastic-serverless-forwarder.html[Elastic Serverless Forwarder].
 
 [discrete]
 [[firehose-step-four]]
 === Step 4: Send data to the Firehose delivery stream
 
-You can configure a variety of log sources to send data to Firehose delivery streams. Refer to the https://docs.aws.amazon.com/firehose/latest/dev/basic-write.html[AWS documentation] for more information.
-Several services support writing data directly to delivery streams, including Cloudwatch logs. Alternatively, you can also use https://aws.amazon.com/dms/[AWS Database Migration Service (DMS)] to create streaming data pipelines to Firehose.
-For example, a typical workflow for sending VPC Flow Logs to Firehose would be the following:
+You can configure a variety of log sources to send data to Firehose streams directly for example VPC flow logs.
+Some services don't support publishing logs directly to Firehose but they do support publishing logs to CloudWatch logs, such as CloudTrail and Lambda.
+Refer to the https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.html[AWS documentation] for more information.
+
+For example, a typical workflow for sending CloudTrail logs to Firehose would be the following:
 
-- Publish VPC Flow Logs to a Cloudwatch log group. Refer to the AWS documentation https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html[about publishing flow logs].
-- Create a subscription filter in the CloudWatch log group to the Firehose delivery stream. Refer to the AWS documentation https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample[about using subscription filters].
+- Publish CloudTrail logs to a Cloudwatch log group. Refer to the AWS documentation https://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html[about publishing CloudTrail logs].
+- Create a subscription filter in the CloudWatch log group to the Firehose stream. Refer to the AWS documentation https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample[about using subscription filters].
 
 
 For more information on Amazon Data Firehose, you can also check the https://docs.elastic.co/integrations/awsfirehose[Amazon Data Firehose Integrations documentation].