Provisioned Lambda timeout on Loading Secret Manager

[andi-pangeran]

Need help to make lambda able to read secrets.

Get this error when provisioning lambda from docker

Log:

[run-lambda-entrypoint] time="2023-02-07T02:24:55Z" level=debug msg="Loading Secret Manager entry arn:aws:secretsmanager:ap-southeast-3:*****:secret:***-MpQE8U as environment variables."
[run-lambda-entrypoint] time="2023-02-07T02:25:06Z" level=debug msg="Loading Secret Manager entry arn:aws:secretsmanager:ap-southeast-3:*****:secret:***-MpQE8U as environment variables."
START RequestId: 90f7d618-8fdf-4d96-b8c8-b31cd8a4e348 Version: $LATEST
2023-02-07T02:25:36.085Z 90f7d618-8fdf-4d96-b8c8-b31cd8a4e348 Task timed out after 30.03 seconds

[run-lambda-entrypoint] time="2023-02-07T03:02:21Z" level=error msg="FAIL Loading Secret Manager entry arn:aws:secretsmanager:ap-southeast-3:***:secret:***-MpQE8U fail operation error Secrets Manager: GetSecretValue, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post \"https://secretsmanager.ap-southeast-3.amazonaws.com/\": dial tcp 108.136.159.12:443: i/o timeout."

my configuration:

  name      = "edo-daily-installments-cron"
  image_uri = local.image

  run_in_vpc = true
  vpc_id     = dependency.vpc.outputs.vpc_id
  subnet_ids = dependency.vpc.outputs.private_app_subnet_ids

  environment_variables = merge(
    include.envcommon.locals.environment_variables,
    {
      IMAGE_URI                       = local.image,
      SECRETS_MANAGER_ARN             = local.edo_cron_workers_secrets_manager_arn,
    }
  )

  iam_policy = {
    SecretsAccess = {
      actions = [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds",
        "secretsmanager:PutSecretValue",
        "secretsmanager:UpdateSecret",
        "secretsmanager:TagResource",
        "secretsmanager:UntagResource"
      ],
      resources = ["${local.edo_cron_workers_secrets_manager_arn}"]
      effect    = "Allow"
    }
  }

  cloudwatch_log_group_retention_in_days = 14

seems like related to https://aws.amazon.com/blogs/security/how-to-connect-to-aws-secrets-manager-service-within-a-virtual-private-cloud/

Thank you

---

Tracked in ticket #109873

[andi-pangeran]

Solved:

steps:

1. create VPC endpoint

terraform {
  source = "git::[email protected]:gruntwork-io/terraform-aws-vpc.git//modules/vpc-interface-endpoint?ref=v0.22.4"
}

inputs = {
  vpc_id             = dependency.vpc.outputs.vpc_id
  subnet_ids         = dependency.vpc.outputs.private_app_subnet_ids

  create_https_security_group = true
  enable_secretsmanager_endpoint = true
}

2. make sure enable "should_create_outbound_rule" on lambda provisioning

  run_in_vpc = true
  vpc_id     = dependency.vpc.outputs.vpc_id
  subnet_ids = dependency.vpc.outputs.private_app_subnet_ids
  should_create_outbound_rule = true