Add a new section on how to collect CloudTrail events using Amazon Da…

…ta Firehose (#3823) (#3884)

* Remove screenshots and adjust text accordingly

* Test the structure

* Add more steps

* Add draft content from public notes

* Fix text and structure 1

* Fix images

* Fix text and structure 2

* Fix text and structure 3

* Fix text and structure 4

* Fix text and structure 5

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Fix headings

* Integrate reviewer's feedback

* Remove link

* Update step 1

---------

Co-authored-by: Maurizio Branca <[email protected]>
Co-authored-by: Brandon Morelli <[email protected]>
(cherry picked from commit ec22304)

# Conflicts:
#	docs/en/observability/cloud-monitoring/aws/monitor-aws-agent.asciidoc

Co-authored-by: Arianna Laudazzi <[email protected]>
Co-authored-by: Brandon Morelli <[email protected]>

docs/en/observability/cloud-monitoring/aws/monitor-amazon-intro.asciidoc

@@ -36,6 +36,8 @@ include::monitor-aws-firehose.asciidoc[]
 
 include::monitor-aws-vpc-flow-logs.asciidoc[leveloffset=+2]
 
+include::monitor-aws-cloudtrail-firehose.asciidoc[leveloffset=+2]
+
 include::monitor-aws-firehose-troubleshooting.asciidoc[leveloffset=+2]
 
 include::monitor-aws-esf.asciidoc[]

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

@@ -0,0 +1,203 @@
+[[monitor-aws-cloudtrail-firehose]]
+= Monitor CloudTrail logs
+
+++++
+<titleabbrev>Monitor CloudTrail logs</titleabbrev>
+++++
+
+In this section, you'll learn how to monitor and analyze the CloudTrail logs you send to Elastic with Amazon Data Firehose. You will go through the following steps:
+
+- Install AWS integration in {kib}
+- Export Cloudtrail events to CloudWatch
+- Set up a Firehose delivery stream
+- Set up a subscription filter to route Cloudtrail events to a delivery stream
+- Visualize your CloudTrail logs in {kib}
+
+
+[discrete]
+[[firehose-cloudtrail-prerequisites]]
+== Before you begin
+
+We assume that you already have:
+
+- An AWS account with permissions to pull the necessary data from AWS.
+- A deployment using our hosted {ess} on {ess-trial}[{ecloud}]. The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data. AWS Data Firehose works with Elastic Stack version 7.17 or greater, running on Elastic Cloud only.
+
+IMPORTANT: Make sure the deployment is on AWS, because the Amazon Data Firehose delivery stream connects specifically to an endpoint that needs to be on AWS.
+
+[discrete]
+[[firehose-cloudtrail-step-one]]
+== Step 1: Install AWS integration in {kib}
+
+. In {kib}, navigate to *Management* > *Integrations* and browse the catalog to find the Amazon Data Firehose integration.
+
+. Navigate to the *Settings* tab and click *Install Amazon Data Firehose assets*.
+
+[discrete]
+[[firehose-cloudtrail-step-two]]
+== Step 2: Export Cloudtrail events to CloudWatch
+
+image::firehose-cloudtrail-cloudwatch.png[Cloudtrail to CloudWatch]
+
+To export CloudTrail logs to CloudWatch, you must set up a *trail* through the following steps:
+
+. Go to the https://console.aws.amazon.com/[AWS console] and navigate to CloudTrail.  
+
+. Click *Create trail* and configure the general details on the *Choose trail attributes* panel, like:
++
+* Trail name
+* Storage location
++
+By default, CloudTrail exports data to an S3 bucket. It isn’t possible to opt-out from S3.
+
+. Specify the encryption options.
++
+When exporting data from CloudTrail to S3, it is recommended to enable
+*Log file SSE-KMS encryption*. You can use an existing AWS KMS key, or create a new one.
+
+. Enable *CloudWatch Logs* and confirm the *Log group name*.
++
+CloudTrail offers the option to send events to CloudWatch as logs. You
+must enable this option to forward the events to Amazon Data Firehose.
++
+You also need to create an IAM Role, or select an existing one, to enable CloudTrail to put log events into a CloudWatch stream.
+
+. From the *Choose log events* panel, select the event types you want to send to Elastic.
+
+. Review the attributes and log events you have specified in the previous steps and click *Create trail*.
+
+. Verify everything is working as expected.
++
+Open the log group you just created on CloudWatch and make sure there are events from the CloudTrail you have just created.
++
+image::firehose-verify-events-cloudwatch.png[Verify events in CloudWatch]
+
+[discrete]
+[[firehose-cloudtrail-step-three]]
+== Step 3: Set up a Firehose delivery stream
+
+image::firehose-delivery-stream.png[Firehose delivery stream]
+
+You now have a CloudWatch log group with events coming from CloudTrail.
+For more information on how to set up a Amazon Data Firehose delivery stream to send data to Elastic Cloud, you can also check the <<monitor-aws-firehose,setup guide>>.
+
+. Collect {es} endpoint and API key from your deployment on Elastic Cloud.
++
+- Elasticsearch endpoint URL: Enter the Elasticsearch endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*.
+- API key: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.
+
+. Set up the delivery stream by specifying the following data:
++
+- Elastic endpoint URL
+- API key
+- Content encoding: gzip
+- Retry duration: 60 (default) 
+- Backup settings: failed data only to s3 bucket
+
+You now have an Amazon Data Firehose delivery specified with:
+
+- source: direct put 
+- destination: elastic 
+- parameters: es_datastream_name: logs-aws.cloudtrail-default
+
+[discrete]
+[[firehose-cloudtrail-step-four]]
+== Step 4: Set up a subscription filter to route Cloudtrail events to a delivery stream
+
+image::firehose-subscription-filter.png[Firehose subscription filter]
+
+The Amazon Data Firehose delivery stream is ready to send logs to your Elastic Cloud deployment. 
+
+. Visit the log group with the CloudTrail events.
++
+Open the log group where the CloudTrail service is sending the
+events. You must forward these events to an Elastic stack using the
+Amazon Data Firehose delivery stream. CloudWatch log group offers a
+https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html[subscription filter] that allows you to choose log events from the log group and forward them to other services like Amazon Kinesis stream, an Amazon Data Firehose stream, or AWS Lambda.
+
+. Create a subscription filter for Amazon Data Firehose by following these steps.
+
+.. Choose the destination account.
++
+Select the delivery stream you created in step 3.
+
+.. Grant permission.
++
+Follow these steps to enable the CloudWatch service to send log events to the delivery stream in Amazon Data Firehose:
+
+... Create a new role with a trust policy that allows CloudWatch to assume the role.
++
+[source,json]
+----
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Principal": {
+                "Service": "logs.eu-north-1.amazonaws.com"
+            },
+            "Action": "sts:AssumeRole",
+            "Condition": {
+                "StringLike": {
+                    "aws:SourceArn": "arn:aws:logs:eu-north-1:<YOUR ACCOUNT ID>:*"
+                }
+            }
+        }
+    ]
+}
+----
+
+... Assign a new IAM policy to the role that permits ”putting records” into a
+in Amazon Data Firehose delivery stream.
++
+[source,json]
+----
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": "firehose:PutRecord",
+            "Resource": "arn:aws:firehose:eu-north-1:<YOUR ACCOUNT ID>:deliverystream/mbranca-dev-cloudtrail-logs"
+        }
+    ]
+}
+----
+
+When the new role is ready, you can select it in the subscription filter. Select *Amazon CloudTrail* in the log format option to configure log format and filters.
+
+[discrete]
+=== Verify
+
+To check if there are destination error logs, go to the AWS console, visit your Amazon Data Firehose delivery stream, and check for entries in the *Destination error logs*.
+
+If everything is correct, this list should be empty. If there’s an
+error, you can check the details. The following example shows a delivery stream that fails to send records to the Elastic stack due to bad authentication settings:
+
+image::firehose-failed-delivery-stream.png[Firehose failed delivery stream]
+
+The Amazon Data Firehose delivery stream reports the number of failed deliveries and failure details.
+
+[discrete]
+[[firehose-cloudtrail-step-five]]
+== Step 5: Visualize your CloudTrail logs in {kib}
+
+With the new subscription filter running, CloudWatch starts routing new
+CloudTrail log events to the Firehose delivery stream.
+
+image::firehose-monitor-cloudtrail-logs.png[Firehose monitor CloudTrail logs]
+
+Navigate to {kib} and choose among the following monitoring options:
+
+- *Visualize your logs with Discover*
++
+image::firehose-cloudtrail-discover.png[Visualize CloudTrail logs with Disocver]
+
+- *Visualize your logs with Logs explorer*
++
+image::firehose-cloudtrail-logsexplorer.png[Visualize CloudTrail logs with Logs explorer]
+
+- *Visualize your logs with the CloudTrail Dashboard*
++
+image::firehose-cloudtrail-dashboard.png[Visualize CloudTrail logs with CloudTrail Dashboard]