Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combined permissions of Fargate logging into single step #895

Merged
merged 3 commits into from
Feb 25, 2025
Merged

Combined permissions of Fargate logging into single step #895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9a9b0fc
Combined permissions of Fargate logging into single step
fincd-aws Feb 25, 2025
00a00cb
Combined fargate logging steps properly
fincd-aws Feb 25, 2025
4885b57
Consistency edits for Update fargate-logging.adoc
pgasca Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 35 additions & 24 deletions latest/ug/nodes/fargate-logging.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,6 @@ You can also use Amazon Kinesis Data Streams for your log destination. If you us
====
[role="tablist"]
CloudWatch::
*To create a `ConfigMap` for CloudWatch*

+
You have two output options when using CloudWatch:
+
Expand Down Expand Up @@ -166,15 +164,8 @@ data:
----
kubectl apply -f aws-logging-cloudwatch-configmap.yaml
----
.. Download the CloudWatch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json[view the policy] on GitHub.
+
[source,bash,subs="verbatim,attributes"]
----
curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json
----

Amazon OpenSearch Service::
*To create a `ConfigMap` for Amazon OpenSearch Service*
+
If you want to send logs to Amazon OpenSearch Service, you can use https://docs.fluentbit.io/manual/v/1.5/pipeline/outputs/elasticsearch[es] output, which is a plugin written in C. The following example shows you how to use the plugin to send logs to OpenSearch.
+
Expand Down Expand Up @@ -206,17 +197,8 @@ data:
----
kubectl apply -f aws-logging-opensearch-configmap.yaml
----
.. Download the OpenSearch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json[view the policy] on GitHub.
+
[source,bash,subs="verbatim,attributes"]
----
curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json
----
+
Make sure that OpenSearch Dashboards' access control is configured properly. The `all_access role` in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the `security_manager` role. You can add the previous mappings by selecting `Menu`, then `Security`, then `Roles`, and then select the respective roles. For more information, see link:tr/premiumsupport/knowledge-center/es-troubleshoot-cloudwatch-logs/[How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?,type="marketing"].

Firehose::
*To create a `ConfigMap` for Firehose*
+
You have two output options when sending logs to Firehose:
+
Expand Down Expand Up @@ -248,20 +230,49 @@ data:
----
kubectl apply -f aws-logging-firehose-configmap.yaml
----
.. Download the Firehose IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json[view the policy] on GitHub.
====

. Set up permissions for the Fargate Pod execution role to send logs to your destination.

.. Download the IAM policy for your destination to your computer.
+
====
[role="tablist"]
CloudWatch::
Download the CloudWatch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json[view the policy] on GitHub.
+
[source,bash,subs="verbatim,attributes"]
----
curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json
----

Amazon OpenSearch Service::
Download the OpenSearch IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json[view the policy] on GitHub.
+
[source,bash,subs="verbatim,attributes"]
----
curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json
----
+
Make sure that OpenSearch Dashboards' access control is configured properly. The `all_access role` in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the `security_manager` role. You can add the previous mappings by selecting `Menu`, then `Security`, then `Roles`, and then select the respective roles. For more information, see link:tr/premiumsupport/knowledge-center/es-troubleshoot-cloudwatch-logs/[How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?,type="marketing"].

Firehose::
Download the Firehose IAM policy to your computer. You can also https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json[view the policy] on GitHub.
+
[source,bash,subs="verbatim,attributes"]
----
curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json
----
====
. Create an IAM policy from the policy file you downloaded in a previous step.

.. Create an IAM policy from the policy file that you downloaded.
+
[source,bash,subs="verbatim,attributes"]
----
aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json
----
. Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace [.replaceable]`111122223333` with your account ID. Replace [.replaceable]`AmazonEKSFargatePodExecutionRole` with your Pod execution role (for more information, see <<fargate-sg-pod-execution-role>>).

.. Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace [.replaceable]`111122223333` with your account ID. Replace [.replaceable]`AmazonEKSFargatePodExecutionRole` with your Pod execution role (for more information, see <<fargate-sg-pod-execution-role>>).
+
[source,bash,subs="verbatim,attributes,quotes"]
----
Expand Down Expand Up @@ -334,7 +345,7 @@ data:
auto_create_group true
----

The logs are in the {aws} Region that the cluster resides in under CloudWatch. The log group name is `[.replaceable]``my-cluster``-fluent-bit-logs` and the Fluent Bit logstream name is `fluent-bit-[.replaceable]``podname``-[.replaceable]``pod-namespace```.
The logs are in CloudWatch in the same {aws} Region as the cluster. The log group name is `[.replaceable]``my-cluster``-fluent-bit-logs` and the Fluent Bit logstream name is `fluent-bit-[.replaceable]``podname``-[.replaceable]``pod-namespace```.

[NOTE]
====
Expand All @@ -349,7 +360,7 @@ The logs are in the {aws} Region that the cluster resides in under CloudWatch. T

Shipping Fluent Bit process logs to CloudWatch requires additional log ingestion and storage costs. To exclude process logs in an existing `ConfigMap` setup, do the following steps.

. Locate the CloudWatch log group automatically created for your Amazon EKS cluster's Fluent Bit process logs after enabling Fargate logging. It follows the format `{cluster_name}-fluent-bit-logs`.
. Locate the CloudWatch log group automatically created for your Amazon EKS cluster's Fluent Bit process logs after enabling Fargate logging. It follows the format `[.replaceable]``my-cluster``-fluent-bit-logs`.
. Delete the existing CloudWatch log streams created for each Pod's process logs in the CloudWatch log group.
. Edit the `ConfigMap` and set `flb_log_cw: "false"`.
. Restart any existing Pods in the cluster.
Expand Down Expand Up @@ -415,4 +426,4 @@ Events:
Warning LoggingDisabled <unknown> fargate-scheduler Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
----

The Pod events are ephemeral with a time period depending on the settings. You can also view a Pod's annotations using `kubectl describe pod [.replaceable]``pod-name```. In the Pod annotation, there is information about whether the logging feature is enabled or disabled and the reason.
The Pod events are ephemeral with a time period depending on the settings. You can also view a Pod's annotations using `kubectl describe pod [.replaceable]``pod-name```. In the Pod annotation, there is information about whether the logging feature is enabled or disabled and the reason.
3 changes: 2 additions & 1 deletion vale/styles/config/vocabularies/EksDocsVocab/accept.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ VPC Reachability Analyzer
reachability
CNIs?
repo
CIDRs?
CIDRs?
Kinesis