From 9a9b0fc7be105a4845ee829b6fad57c4472e1e81 Mon Sep 17 00:00:00 2001 From: Donovan Date: Tue, 25 Feb 2025 09:43:24 -0800 Subject: [PATCH 1/3] Combined permissions of Fargate logging into single step Merge permissions steps 3 & 4 together, move IAM policy download down into this step as a new tabbed section. The config file steps was too long and logically, the `kubectl apply` is the last sub-step, so the permissions were getting skipped by customers. --- latest/ug/nodes/fargate-logging.adoc | 46 ++++++++++++++++++---------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/latest/ug/nodes/fargate-logging.adoc b/latest/ug/nodes/fargate-logging.adoc index 09e512bd..a1f5400c 100644 --- a/latest/ug/nodes/fargate-logging.adoc +++ b/latest/ug/nodes/fargate-logging.adoc @@ -166,12 +166,6 @@ data: ---- kubectl apply -f aws-logging-cloudwatch-configmap.yaml ---- -.. Download the CloudWatch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json[view the policy] on GitHub. -+ -[source,bash,subs="verbatim,attributes"] ----- -curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json ----- Amazon OpenSearch Service:: *To create a `ConfigMap` for Amazon OpenSearch Service* @@ -206,14 +200,6 @@ data: ---- kubectl apply -f aws-logging-opensearch-configmap.yaml ---- -.. Download the OpenSearch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json[view the policy] on GitHub. -+ -[source,bash,subs="verbatim,attributes"] ----- -curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json ----- -+ -Make sure that OpenSearch Dashboards' access control is configured properly. The `all_access role` in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the `security_manager` role. You can add the previous mappings by selecting `Menu`, then `Security`, then `Roles`, and then select the respective roles. For more information, see link:tr/premiumsupport/knowledge-center/es-troubleshoot-cloudwatch-logs/[How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?,type="marketing"]. Firehose:: *To create a `ConfigMap` for Firehose* @@ -248,6 +234,30 @@ data: ---- kubectl apply -f aws-logging-firehose-configmap.yaml ---- +==== + +. Set up permissions for the Fargate Pod execution role to send logs to your destination. +==== +[role="tablist"] +CloudWatch:: +.. Download the CloudWatch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json[view the policy] on GitHub. ++ +[source,bash,subs="verbatim,attributes"] +---- +curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json +---- + +Amazon OpenSearch Service:: +.. Download the OpenSearch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json[view the policy] on GitHub. ++ +[source,bash,subs="verbatim,attributes"] +---- +curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json +---- ++ +Make sure that OpenSearch Dashboards' access control is configured properly. The `all_access role` in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the `security_manager` role. You can add the previous mappings by selecting `Menu`, then `Security`, then `Roles`, and then select the respective roles. For more information, see link:tr/premiumsupport/knowledge-center/es-troubleshoot-cloudwatch-logs/[How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?,type="marketing"]. + +Firehose:: .. Download the Firehose IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json[view the policy] on GitHub. + [source,bash,subs="verbatim,attributes"] @@ -255,13 +265,15 @@ kubectl apply -f aws-logging-firehose-configmap.yaml curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json ---- ==== -. Create an IAM policy from the policy file you downloaded in a previous step. + +.. Create an IAM policy from the policy file that you downloaded. + [source,bash,subs="verbatim,attributes"] ---- aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json ---- -. Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace [.replaceable]`111122223333` with your account ID. Replace [.replaceable]`AmazonEKSFargatePodExecutionRole` with your Pod execution role (for more information, see <>). + +.. Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace [.replaceable]`111122223333` with your account ID. Replace [.replaceable]`AmazonEKSFargatePodExecutionRole` with your Pod execution role (for more information, see <>). + [source,bash,subs="verbatim,attributes,quotes"] ---- @@ -415,4 +427,4 @@ Events: Warning LoggingDisabled fargate-scheduler Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found ---- -The Pod events are ephemeral with a time period depending on the settings. You can also view a Pod's annotations using `kubectl describe pod [.replaceable]``pod-name```. In the Pod annotation, there is information about whether the logging feature is enabled or disabled and the reason. \ No newline at end of file +The Pod events are ephemeral with a time period depending on the settings. You can also view a Pod's annotations using `kubectl describe pod [.replaceable]``pod-name```. In the Pod annotation, there is information about whether the logging feature is enabled or disabled and the reason. From 00a00cbdcde52de619620b336d03b0aa87cd75b2 Mon Sep 17 00:00:00 2001 From: Donovan Finch Date: Tue, 25 Feb 2025 10:52:20 -0800 Subject: [PATCH 2/3] Combined fargate logging steps properly --- latest/ug/nodes/fargate-logging.adoc | 11 +++++++---- .../config/vocabularies/EksDocsVocab/accept.txt | 3 ++- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/latest/ug/nodes/fargate-logging.adoc b/latest/ug/nodes/fargate-logging.adoc index a1f5400c..75780468 100644 --- a/latest/ug/nodes/fargate-logging.adoc +++ b/latest/ug/nodes/fargate-logging.adoc @@ -237,10 +237,13 @@ kubectl apply -f aws-logging-firehose-configmap.yaml ==== . Set up permissions for the Fargate Pod execution role to send logs to your destination. + +.. Download the IAM policy for your destination to your computer. ++ ==== [role="tablist"] CloudWatch:: -.. Download the CloudWatch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json[view the policy] on GitHub. +Download the CloudWatch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json[view the policy] on GitHub. + [source,bash,subs="verbatim,attributes"] ---- @@ -248,7 +251,7 @@ curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging- ---- Amazon OpenSearch Service:: -.. Download the OpenSearch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json[view the policy] on GitHub. +Download the OpenSearch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json[view the policy] on GitHub. + [source,bash,subs="verbatim,attributes"] ---- @@ -258,7 +261,7 @@ curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging- Make sure that OpenSearch Dashboards' access control is configured properly. The `all_access role` in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the `security_manager` role. You can add the previous mappings by selecting `Menu`, then `Security`, then `Roles`, and then select the respective roles. For more information, see link:tr/premiumsupport/knowledge-center/es-troubleshoot-cloudwatch-logs/[How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?,type="marketing"]. Firehose:: -.. Download the Firehose IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json[view the policy] on GitHub. +Download the Firehose IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json[view the policy] on GitHub. + [source,bash,subs="verbatim,attributes"] ---- @@ -346,7 +349,7 @@ data: auto_create_group true ---- -The logs are in the {aws} Region that the cluster resides in under CloudWatch. The log group name is `[.replaceable]``my-cluster``-fluent-bit-logs` and the Fluent Bit logstream name is `fluent-bit-[.replaceable]``podname``-[.replaceable]``pod-namespace```. +The logs are in CloudWatch in the same {aws} Region as the cluster. The log group name is `[.replaceable]``my-cluster``-fluent-bit-logs` and the Fluent Bit logstream name is `fluent-bit-[.replaceable]``podname``-[.replaceable]``pod-namespace```. [NOTE] ==== diff --git a/vale/styles/config/vocabularies/EksDocsVocab/accept.txt b/vale/styles/config/vocabularies/EksDocsVocab/accept.txt index ab7ed47a..151c00f3 100644 --- a/vale/styles/config/vocabularies/EksDocsVocab/accept.txt +++ b/vale/styles/config/vocabularies/EksDocsVocab/accept.txt @@ -13,4 +13,5 @@ VPC Reachability Analyzer reachability CNIs? repo -CIDRs? \ No newline at end of file +CIDRs? +Kinesis \ No newline at end of file From 4885b57c2f6a906557a17f57964ed69ea7c4dd5d Mon Sep 17 00:00:00 2001 From: pgasca <87044997+pgasca@users.noreply.github.com> Date: Tue, 25 Feb 2025 14:19:54 -0600 Subject: [PATCH 3/3] Consistency edits for Update fargate-logging.adoc Verified that we are no longer using bolded "To do ___" intros for each tabbed list entry throughout guide. --- latest/ug/nodes/fargate-logging.adoc | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/latest/ug/nodes/fargate-logging.adoc b/latest/ug/nodes/fargate-logging.adoc index 75780468..effeb7ce 100644 --- a/latest/ug/nodes/fargate-logging.adoc +++ b/latest/ug/nodes/fargate-logging.adoc @@ -107,8 +107,6 @@ You can also use Amazon Kinesis Data Streams for your log destination. If you us ==== [role="tablist"] CloudWatch:: -*To create a `ConfigMap` for CloudWatch* - + You have two output options when using CloudWatch: + @@ -168,7 +166,6 @@ kubectl apply -f aws-logging-cloudwatch-configmap.yaml ---- Amazon OpenSearch Service:: -*To create a `ConfigMap` for Amazon OpenSearch Service* + If you want to send logs to Amazon OpenSearch Service, you can use https://docs.fluentbit.io/manual/v/1.5/pipeline/outputs/elasticsearch[es] output, which is a plugin written in C. The following example shows you how to use the plugin to send logs to OpenSearch. + @@ -202,7 +199,6 @@ kubectl apply -f aws-logging-opensearch-configmap.yaml ---- Firehose:: -*To create a `ConfigMap` for Firehose* + You have two output options when sending logs to Firehose: + @@ -364,7 +360,7 @@ The logs are in CloudWatch in the same {aws} Region as the cluster. The log grou Shipping Fluent Bit process logs to CloudWatch requires additional log ingestion and storage costs. To exclude process logs in an existing `ConfigMap` setup, do the following steps. -. Locate the CloudWatch log group automatically created for your Amazon EKS cluster's Fluent Bit process logs after enabling Fargate logging. It follows the format `{cluster_name}-fluent-bit-logs`. +. Locate the CloudWatch log group automatically created for your Amazon EKS cluster's Fluent Bit process logs after enabling Fargate logging. It follows the format `[.replaceable]``my-cluster``-fluent-bit-logs`. . Delete the existing CloudWatch log streams created for each Pod's process logs in the CloudWatch log group. . Edit the `ConfigMap` and set `flb_log_cw: "false"`. . Restart any existing Pods in the cluster.