[AWS] [Lambda] Add Event Source Mapping metrics (elastic#11907)

* add esm metrics

* add esm documentation

* improve dashboard

* bump package version

* improve documentation

* Update packages/aws/_dev/build/docs/lambda.md

Co-authored-by: Agi K Thomas <[email protected]>

* improve documentation

* add toggle for esm metrics

* improve dashboard

* fix y label naming

* update dashboard img

* improve dashboard image

* remove redundant documentation

---------

Co-authored-by: Agi K Thomas <[email protected]>

packages/aws/_dev/build/docs/lambda.md

@@ -13,7 +13,24 @@ For example, you could use this integration to track throttled lambda functions,
 The AWS Lambda integration collects one type of data: metrics.
 
 **Metrics** give you insight into the state of AWS Lambda.
-Metrics collected by the AWS Lambda integration include the number of times your function code is executed, the amount of time that your function code spends processing an event, the number of invocations that result in a function error, and more. See more details in the [Metrics reference](#metrics-reference).
+Metrics collected by the AWS Lambda integration include the number of times your function code is executed, the amount of time that your function code spends processing an event, the number of invocations that result in a function error, and more.
+
+See more details in the [Metrics reference](#metrics-reference).
+
+## Event Source Mapping Metrics
+
+This integration can also collect Event Source Mapping (ESM) metrics, which track how your Lambda function processes records from event sources like Amazon SQS, DynamoDB, or Kinesis. 
+
+To collect these metrics, you must:  
+1. Enable Event Source Mapping for your Lambda function by following the steps outlined in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html#event-source-mapping-metrics).  
+2. Enable the configuration flag `Collect Event Source Mapping metrics`.  
+
+Important notes:
+- Enabling this feature may incur additional costs
+- Not all metrics are available for every event source type
+- Metric collection may be affected by CloudWatch or Lambda service availability
+
+See more details about [Event Source Mapping Metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html#event-source-mapping-metrics).
 
 ## Requirements
 

packages/aws/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.34.0"
+  changes:
+    - description: Add Lambda Event Source Mapping metrics and improve Lambda dashboard to display the new metrics.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11907
 - version: "2.33.0"
   changes:
     - description: Add option to check linked accounts when using log group prefixes to derive matching log groups

packages/aws/data_stream/lambda/agent/stream/stream.yml.hbs

@@ -45,6 +45,7 @@ tags_filter: {{tags_filter}}
 {{#if proxy_url }}
 proxy_url: {{proxy_url}}
 {{/if}}
+{{#if collect_esm_metrics}}
 metrics:
 - namespace: AWS/Lambda
   resource_type: lambda
@@ -78,3 +79,45 @@ metrics:
   - DeadLetterErrors
   - DestinationDeliveryFailures
   - AsyncEventsReceived
+  - PolledEventCount
+  - FilteredOutEventCount
+  - InvokedEventCount
+  - FailedInvokeEventCount
+  - DroppedEventCount
+  - OnFailureDestinationDeliveredEventCount
+  - DeletedEventCount
+{{ else }}
+metrics:
+- namespace: AWS/Lambda
+  resource_type: lambda
+  statistic: ["Average"]
+  name:
+  - Invocations
+  - Errors
+  - DeadLetterErrors
+  - DestinationDeliveryFailures
+  - Duration
+  - Throttles
+  - IteratorAge
+  - ConcurrentExecutions
+  - UnreservedConcurrentExecutions
+  - AsyncEventAge
+- namespace: AWS/Lambda
+  resource_type: lambda
+  statistic: ["Maximum"]
+  name:
+  - ProvisionedConcurrentExecutions
+  - ProvisionedConcurrencyUtilization
+- namespace: AWS/Lambda
+  resource_type: lambda
+  statistic: ["Sum"]
+  name:
+  - ProvisionedConcurrencyInvocations
+  - ProvisionedConcurrencySpilloverInvocations
+  - Invocations
+  - Throttles
+  - Errors
+  - DeadLetterErrors
+  - DestinationDeliveryFailures
+  - AsyncEventsReceived
+{{/if}}

packages/aws/data_stream/lambda/fields/fields.yml

@@ -16,6 +16,10 @@
           type: keyword
           dimension: true
           description: Use the ExecutedVersion dimension to compare error rates for two versions of a function that are both targets of a weighted alias.
+        - name: EventSourceMappingUUID
+          type: keyword
+          dimension: true
+          description: The identifier of the event source mapping.
     - name: lambda
       type: group
       fields:
@@ -103,6 +107,34 @@
               unit: ms
               metric_type: gauge
               description: The average time between when Lambda successfully queues the event and when the function is invoked.
+            - name: PolledEventCount.sum
+              metric_type: gauge
+              type: double
+              description: The number of events that Lambda reads successfully from the event source.
+            - name: FilteredOutEventCount.sum
+              metric_type: gauge
+              type: double
+              description: For event source mapping with a filter criteria, the number of events excluded based on the specified criteria.
+            - name: InvokedEventCount.sum
+              metric_type: gauge
+              type: double
+              description: The number of events that invoked your Lambda function. Use this metric to verify that events are properly invoking your function.
+            - name: FailedInvokeEventCount.sum
+              metric_type: gauge
+              type: double
+              description: The number of events that Lambda tried to invoke your function with, but failed.
+            - name: DroppedEventCount.sum
+              metric_type: gauge
+              type: double
+              description: The number of events that Lambda dropped due to expiry or retry exhaustion.
+            - name: OnFailureDestinationDeliveredEventCount.sum
+              metric_type: gauge
+              type: double
+              description: For event source mappings with an on-failure destination configured, the number of events sent to that destination.
+            - name: DeletedEventCount.sum
+              metric_type: gauge
+              type: double
+              description: The number of events that Lambda successfully deletes after processing.
     - name: cloudwatch
       type: group
       fields:

packages/aws/data_stream/lambda/manifest.yml

@@ -55,5 +55,13 @@ streams:
         required: false
         show_user: false
         description: Accepts an AWS account ID linked to the monitoring account. Works only if include_linked_accounts is set to true. If set, monitoring data will only include data from the given account.
+      - name: collect_esm_metrics
+        type: bool
+        title: Collect Event Source Mapping metrics
+        multi: false
+        required: true
+        show_user: true
+        default: false
+        description: Follow the [ESM metrics setup instructions](https://www.elastic.co/docs/current/integrations/aws/lambda#event-source-mapping-metrics) before enabling this feature.
     title: AWS Lambda metrics
     description: Collect AWS Lambda metrics

packages/aws/docs/lambda.md

@@ -13,7 +13,24 @@ For example, you could use this integration to track throttled lambda functions,
 The AWS Lambda integration collects one type of data: metrics.
 
 **Metrics** give you insight into the state of AWS Lambda.
-Metrics collected by the AWS Lambda integration include the number of times your function code is executed, the amount of time that your function code spends processing an event, the number of invocations that result in a function error, and more. See more details in the [Metrics reference](#metrics-reference).
+Metrics collected by the AWS Lambda integration include the number of times your function code is executed, the amount of time that your function code spends processing an event, the number of invocations that result in a function error, and more.
+
+See more details in the [Metrics reference](#metrics-reference).
+
+## Event Source Mapping Metrics
+
+This integration can also collect Event Source Mapping (ESM) metrics, which track how your Lambda function processes records from event sources like Amazon SQS, DynamoDB, or Kinesis. 
+
+To collect these metrics, you must:  
+1. Enable Event Source Mapping for your Lambda function by following the steps outlined in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html#event-source-mapping-metrics).  
+2. Enable the configuration flag `Collect Event Source Mapping metrics`.  
+
+Important notes:
+- Enabling this feature may incur additional costs
+- Not all metrics are available for every event source type
+- Metric collection may be affected by CloudWatch or Lambda service availability
+
+See more details about [Event Source Mapping Metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html#event-source-mapping-metrics).
 
 ## Requirements
 
@@ -127,6 +144,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 | @timestamp | Event timestamp. | date |  |  |
 | agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |  |  |
 | aws.cloudwatch.namespace | The namespace specified when query cloudwatch api. | keyword |  |  |
+| aws.dimensions.EventSourceMappingUUID | The identifier of the event source mapping. | keyword |  |  |
 | aws.dimensions.ExecutedVersion | Use the ExecutedVersion dimension to compare error rates for two versions of a function that are both targets of a weighted alias. | keyword |  |  |
 | aws.dimensions.FunctionName | Lambda function name. | keyword |  |  |
 | aws.dimensions.Resource | Resource name. | keyword |  |  |
@@ -135,14 +153,21 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 | aws.lambda.metrics.ConcurrentExecutions.avg | The average number of function instances that are processing events. | double |  | gauge |
 | aws.lambda.metrics.DeadLetterErrors.avg | For asynchronous invocation, the average number of times Lambda attempts to send an event to a dead-letter queue but fails. | double |  | gauge |
 | aws.lambda.metrics.DeadLetterErrors.sum | For asynchronous invocation, the total number of times Lambda attempts to send an event to a dead-letter queue but fails. | double |  | gauge |
+| aws.lambda.metrics.DeletedEventCount.sum | The number of events that Lambda successfully deletes after processing. | double |  | gauge |
 | aws.lambda.metrics.DestinationDeliveryFailures.avg | For asynchronous invocation, the average number of times Lambda attempts to send an event to a destination but fails. | double |  | gauge |
 | aws.lambda.metrics.DestinationDeliveryFailures.sum | For asynchronous invocation, the total number of times Lambda attempts to send an event to a destination but fails. | double |  | gauge |
+| aws.lambda.metrics.DroppedEventCount.sum | The number of events that Lambda dropped due to expiry or retry exhaustion. | double |  | gauge |
 | aws.lambda.metrics.Duration.avg | The average amount of time that your function code spends processing an event. | double |  | gauge |
 | aws.lambda.metrics.Errors.avg | The average number of invocations that result in a function error. | double |  | gauge |
 | aws.lambda.metrics.Errors.sum | The total number of invocations that result in a function error. | double |  | gauge |
+| aws.lambda.metrics.FailedInvokeEventCount.sum | The number of events that Lambda tried to invoke your function with, but failed. | double |  | gauge |
+| aws.lambda.metrics.FilteredOutEventCount.sum | For event source mapping with a filter criteria, the number of events excluded based on the specified criteria. | double |  | gauge |
 | aws.lambda.metrics.Invocations.avg | The average number of times your function code is executed, including successful executions and executions that result in a function error. | double |  | gauge |
 | aws.lambda.metrics.Invocations.sum | The total number of times your function code is executed, including successful executions and executions that result in a function error. | double |  | gauge |
+| aws.lambda.metrics.InvokedEventCount.sum | The number of events that invoked your Lambda function. Use this metric to verify that events are properly invoking your function. | double |  | gauge |
 | aws.lambda.metrics.IteratorAge.avg | For event source mappings that read from streams, the average age of the last record in the event. | double |  | gauge |
+| aws.lambda.metrics.OnFailureDestinationDeliveredEventCount.sum | For event source mappings with an on-failure destination configured, the number of events sent to that destination. | double |  | gauge |
+| aws.lambda.metrics.PolledEventCount.sum | The number of events that Lambda reads successfully from the event source. | double |  | gauge |
 | aws.lambda.metrics.ProvisionedConcurrencyInvocations.sum | The total number of times your function code is executed on provisioned concurrency. | long |  | gauge |
 | aws.lambda.metrics.ProvisionedConcurrencySpilloverInvocations.sum | The total number of times your function code is executed on standard concurrency when all provisioned concurrency is in use. | long |  | gauge |
 | aws.lambda.metrics.ProvisionedConcurrencyUtilization.max | For a version or alias, the maximum value of ProvisionedConcurrentExecutions divided by the total amount of provisioned concurrency allocated. | long |  | gauge |