Deploying Cumulus 18.5.1 Notes + Manual changes

[wphyojpl]

• Most changes are done in unity-uds-terraform and cumulus-deploy terraform which are pushed to github.
• One of them is upgrading to Java11 for cnm_to_cma_task and cnm_response_task lambdas.
• It is done by upgrading the version.
• unity-uds-terraform is looking for SNS which doesn't exist in a fresh install.
• It is fixed by passing SNS arn straight away instead of SNS name for unity-uds to look for the resource to get the arn.

There are 3 changes which are done directly in cumulus terraform

• Error 1:

│ Error: creating IAM Role (nnn-nnn-nnnnnnn-dla-migration): AccessDenied: User: arn:aws:sts::xxxxxxxxxxxxx:assumed-role/mcp-tenantOperator/user_x is not authorized to perform: 
iam:CreateRole on resource: arn:aws:iam::xxxxxxxxxxxxx:role/nnn-nnn-nnnnnnn-dla-migration because no identity-based policy allows the iam:CreateRole action
│       status code: 403, request id: 11a856d2-d757-459d-8ca7-ef701a9c0e3f
│ 
│   with module.cumulus.module.dla_migration_lambda.aws_iam_role.dla_migration_role,
│   on .terraform/modules/cumulus/lambdas/dla-migration/iam.tf line 27, in resource "aws_iam_role" "dla_migration_role":
│   27: resource "aws_iam_role" "dla_migration_role" {

permissions_boundary has to be hardcoded to "arn:aws:iam::xxxxxxxxxxxxx:policy/mcp-tenantOperator-AMI-APIG" to fix that
in /cumulus-tf/.terraform/modules/cumulus/lambdas/dla-migration/iam.tf

resource "aws_iam_role" "dla_migration_role" {
  name                 = "${var.prefix}-dla-migration"
  assume_role_policy   = data.aws_iam_policy_document.dla_migration_assume_role_policy.json
  permissions_boundary = var.permissions_boundary_arn
  tags                 = var.tags
}

• Error 2:

│ Error: invoking Lambda Function (nnn-nnn-nnnnnnn-CustomBootstrap): returned error: "{"errorType":"ResponseError","errorMessage":"Response Error","trace":["ResponseError: Response Error","    
at IncomingMessage.<anonymous> (/var/task/es-client/node_modules/@elastic/elasticsearch/lib/Transport.js:289:1)","    
at IncomingMessage.emit (node:events:531:35)","    at endReadableNT (node:internal/streams/readable:1696:12)","    
at processTicksAndRejections (node:internal/process/task_queues:82:21)"]}"
│ 
│   with module.cumulus.module.archive.data.aws_lambda_invocation.custom_bootstrap[0],
│   on .terraform/modules/cumulus/tf-modules/archive/bootstrap.tf line 29, in data "aws_lambda_invocation" "custom_bootstrap":
│   29: data "aws_lambda_invocation" "custom_bootstrap" {

Cloudwatch logs:

2024-11-19T17:13:56.114Z 8c904e0c-bfc9-4cb0-a0ec-8c602f11b09c ERROR Invoke Error {"errorType":"ResponseError","errorMessage":"Response Error","name":"ResponseError","meta":
  {"body":{"Message":"User: arn:aws:sts::xxxxxxxxxxxxx:assumed-role/nnn-nnn-nnnnnnn-lambda-processing/nnn-nnn-nnnnnnn-CustomBootstrap is not authorized to perform: 
    es:ESHttpPut because no identity-based policy allows the es:ESHttpPut action"},"statusCode":403,"headers":{"date":"Tue, 19 Nov 2024 17:13:55 GMT","content-type":"application/json","content-length":"231","connection":"keep-alive","x-amzn-requestid":"2ba60944-533a-4401-82c0-880b8c92e0c7","access-control-allow-origin":"*"},"warnings":null,"meta":{"context":null,"request":{"params":{"method":"PUT","path":"/_cluster/settings","body":"{\"persistent\":{\"action.auto_create_index\":false}}","querystring":"","headers":{"User-Agent":"elasticsearch-js/5.6.22 (linux 5.10.227-239.884.amzn2.x86_64-x64; Node.js v20.18.0)","Content-Type":"application/json","Content-Length":"49"},"timeout":50000},"options":{"warnings":null},"id":1},"name":"elasticsearch-js","connection":
  {"url":"https://vpc-nnn-nnn-nnnnnnn-es-vpc-zzzzzzzzzzzzzzzzzzzzzzz.us-west-2.es.amazonaws.com/",
    "id":"https://vpc-nnn-nnn-nnnnnnn-es-vpc-zzzzzzzzzzzzzzzzzzzzzzz.us-west-2.es.amazonaws.com/","headers":null,"deadCount":0,"resurrectTimeout":0,"_openRequests":0,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}},"stack":["ResponseError: Response Error"," at IncomingMessage.<anonymous> (/var/task/es-client/node_modules/@elastic/elasticsearch/lib/Transport.js:289:1)"," at IncomingMessage.emit (node:events:531:35)"," at endReadableNT (node:internal/streams/readable:1696:12)"," at processTicksAndRejections (node:internal/process/task_queues:82:21)"]}

Adding the following in /cumulus-tf/.terraform/modules/cumulus/tf-modules/cumulus/iam.tf, but it did not help.

  actions =[
    "es:ESHttp*"
  ]
  resources = ["*"]
}

The following access policy is also added to Cumulus Opensearch

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxxx:domain/uds-dev-cumulus-es-vpc/*"
    }
  ]
}

[wphyojpl]

New error while creating a creation

{
    "error": {
        "name": "error",
        "message": "insert into \"collections\" (\"created_at\", \"duplicate_handling\", \"files\", \"granule_id_extraction_regex\", \"granule_id_validation_regex\", \"ignore_files_config_for_discovery\", \"meta\", \"name\", \"process\", \"report_to_ems\", \"sample_file_name\", \"tags\", \"updated_at\", \"url_path\", \"version\") values ($1, $2, $3, $4, $5, DEFAULT, DEFAULT, $6, $7, $8, $9, DEFAULT, $10, $11, $12) returning * - relation \"collections\" does not exist",
        "stack": [
            "error: insert into \"collections\" (\"created_at\", \"duplicate_handling\", \"files\", \"granule_id_extraction_regex\", \"granule_id_validation_regex\", \"ignore_files_config_for_discovery\", \"meta\", \"name\", \"process\", \"report_to_ems\", \"sample_file_name\", \"tags\", \"updated_at\", \"url_path\", \"version\") values ($1, $2, $3, $4, $5, DEFAULT, DEFAULT, $6, $7, $8, $9, DEFAULT, $10, $11, $12) returning * - relation \"collections\" does not exist",
            "    at Parser.parseErrorMessage (/var/task/db/node_modules/pg-protocol/dist/parser.js:283:1)",
            "    at Parser.handlePacket (/var/task/db/node_modules/pg-protocol/dist/parser.js:122:1)",
            "    at Parser.parse (/var/task/db/node_modules/pg-protocol/dist/parser.js:35:1)",
            "    at TLSSocket.<anonymous> (/var/task/db/node_modules/pg-protocol/dist/index.js:11:1)",
            "    at TLSSocket.emit (node:events:519:28)",
            "    at addChunk (node:internal/streams/readable:559:12)",
            "    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)",
            "    at TLSSocket.Readable.push (node:internal/streams/readable:390:5)",
            "    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)"
        ]
    },
    "level": "error",
    "message": "Error occurred while trying to create collection: {\"reportToEms\":false,\"duplicateHandling\":\"skip\",\"granuleId\":\"^test_file.*$\",\"granuleIdExtraction\":\"(^test_file.*)(\\\\.nc|\\\\.nc\\\\.cas|\\\\.cmr\\\\.xml)\",\"process\":\"stac\",\"name\":\"URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:TEST2\",\"version\":\"2403201440\",\"url_path\":\"^test_file.*\\\\.nc$\",\"sampleFileName\":\"test_file01.nc\",\"files\":[{\"reportToEms\":false,\"type\":\"data\",\"sampleFileName\":\"test_file01.nc\",\"bucket\":\"protected\",\"regex\":\"^test_file.*\\\\.nc$\"},{\"reportToEms\":false,\"type\":\"metadata\",\"sampleFileName\":\"test_file01.nc.cas\",\"bucket\":\"protected\",\"regex\":\"^test_file.*\\\\.nc.cas$\"},{\"reportToEms\":false,\"type\":\"metadata\",\"sampleFileName\":\"test_file01.nc.cmr.xml\",\"bucket\":\"protected\",\"regex\":\"^test_file.*\\\\.nc.cmr.xml$\"},{\"reportToEms\":false,\"type\":\"metadata\",\"sampleFileName\":\"test_file01.nc.stac.json\",\"bucket\":\"protected\",\"regex\":\"^test_file.*\\\\.nc.stac.json$\"}],\"updatedAt\":1732061375428,\"createdAt\":1732061375428}",
    "sender": "@cumulus/api/collections",
    "timestamp": "2024-11-20T00:09:35.566Z"
}