CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

jennyhliu · 2025-01-21T23:37:46Z

Summary: Summary of changes

Addresses CUMULUS-3960: Update CMR metadata with new collection info

Changes

Updated PostToCmr task to be able to republish granules

Testing

Verified that PostToCmr can republish granule:

Added a workflow with the RepublishTest:

    "RepublishTest": {
    "Parameters": {
      "cma": {
        "event.$": "$",
        "ReplaceConfig": {
          "FullMessage": true
        },
        "task_config": {
          "bucket": "{$.meta.buckets.internal.name}",
          "stack": "{$.meta.stack}",
          "cmr": "{$.meta.cmr}",
          "launchpad": "{$.meta.launchpad}",
          "etags": "{$.meta.file_etags}",
          "concurrency": 10,
          "republish": true
        }
      }
    },
    "Type": "Task",
    "Resource": "${post_to_cmr_task_arn}",
    "Next": "BackupGranulesToLzards",
    "Retry": [
      {
        "ErrorEquals": [
          "Lambda.ServiceException",
          "Lambda.AWSLambdaException",
          "Lambda.SdkClientException"
        ],
        "IntervalSeconds": 2,
        "MaxAttempts": 6,
        "BackoffRate": 2
      }
    ]
  },

Updated spec/parallel/orca/OrcaBackupAndRecoverySpec.js to ingest a granule only (xdescribe any tests except S3 Ingest Granules, comment out the cleanup code)
Got the output cumulus message in s3 from IngestAndPublishGranuleWithOrca workflow execution.
Manually updated .cmr.xml in s3 to have collection ShortName MOD09GQ-> MOD09GQ-AZ
Ran the workflow with the cumulus message we got from IngestAndPublishGranuleWithOrca with etags removed. We can update etags as well.
Verified CMR granule has updated collection ShortName: https://cmr.uat.earthdata.nasa.gov/search/granules.umm_json?provider=CUMULUS&granule_ur=MOD09GQ.A5513498.Nji_76.006.6698559176437
Note: The cmr concept-id remains the same in my test

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

This reverts commit 4f790ff.

This reverts commit 548fbf1.

jennyhliu · 2025-01-24T18:45:44Z

.eslintrc.js

@@ -62,6 +62,7 @@ module.exports = {
  },
  globals: {
    JSX: true,
+    AggregateError: false,


This is to resolve eslint error error 'AggregateError' is not defined. Later versions of eslint have resolved the error.

jennyhliu · 2025-01-27T13:15:02Z

tasks/post-to-cmr/index.js

+  const results = await pMap(
+    updatedCMRFiles,
+    (cmrFile) => publish2CMR(cmrFile, cmrSettings, cmrRevisionId),
+    { concurrency }


stopOnError: true is default, if set to false, it will change current lambda behavior

can this even be set by a user or someone? it doesnt seem defined anyways

jennyhliu · 2025-01-28T14:34:32Z

tasks/post-to-cmr/.nycrc.json

@@ -5,6 +5,6 @@
  ],
  "statements": 94.0,
  "functions": 88.0,
-  "branches": 97.0,
+  "branches": 88.0,


Only the handler function is not covered as before, not sure how to improve this.

…o jl/CUMULUS-3960

Nnaga1

Just some questions/comments I had, one or two small NITs, once answered + some local testing I'll approve 🙌

Nnaga1 · 2025-01-30T21:10:44Z

packages/cmrjs/src/cmr-utils.js

@@ -35,6 +35,16 @@ const {

 const log = new Logger({ sender: '@cumulus/cmrjs/src/cmr-utils' });

+/**
+ * @typedef {{


👍 better than having a {provider, clientId, username, password....} everywhere

Nnaga1 · 2025-01-30T21:11:20Z

packages/cmrjs/src/cmr-utils.js

@@ -233,6 +243,17 @@ async function publish2CMR(cmrPublishObject, creds, cmrRevisionId) {
  throw new Error(`invalid cmrPublishObject passed to publis2CMR ${JSON.stringify(cmrPublishObject)}`);
 }

+/**


is there a reason why this wasn't here before? did we never have the case of having to remove from CMR until now or was there just another way to do it (via API or something)

mainly for my curiousity I guess

Nnaga1 · 2025-01-30T21:14:31Z

tasks/post-to-cmr/README.md

@@ -25,6 +25,8 @@ Config object fields:
 | process | string | (required) | Process the granules went through
 | stack | string | (required) | Name of deployment stack
 | cmr | object | (required) | CMR credentials object
+| concurrency | number | 20 | Maximum concurrency of requests to CMR


is 20/false the default, maybe you could say:

| concurrency | (optional) | Maximum concurrency of requests to CMR, defaults to 20

or something like that, just a nit tho

The other task puts default value to this column.
https://github.com/nasa/cumulus/blob/master/tasks/files-to-granules/README.md

Nnaga1 · 2025-01-30T21:18:15Z

tasks/post-to-cmr/index.js

+ * @throws {Error} - Error from CMR request
+ */
+async function removeGranuleFromCmr({ granules, cmrSettings, concurrency }) {
+  const granulesToUnpublish = granules.filter((granule) => granule.published || !!granule.cmrLink);


its considered published if it doesnt have a granule.cmrLink ?

In theory published granule should have cmrLink, but we should be able to delete the granule from cmr if we don't have the link

Nnaga1 · 2025-01-30T21:21:12Z

tasks/post-to-cmr/index.js

+  const granulesToUnpublish = granules.filter((granule) => granule.published || !!granule.cmrLink);
+  await pMap(
+    granulesToUnpublish,
+    (granule) => removeFromCMR(granule.granuleId, cmrSettings),


Nnaga1 · 2025-01-30T21:24:08Z

tasks/post-to-cmr/index.js

+  const results = await pMap(
+    updatedCMRFiles,
+    (cmrFile) => publish2CMR(cmrFile, cmrSettings, cmrRevisionId),
+    { concurrency }


can this even be set by a user or someone? it doesnt seem defined anyways

Nnaga1 · 2025-01-30T21:27:45Z

tasks/post-to-cmr/tests/cmr-test.js

+    cmrClient.CMR.prototype.ingestGranule.restore();
+  });
+
+  await s3PutObject({


I get the other calls/functions but what does this s3PutObject do in the case of the test?

jennyhliu added 2 commits January 21, 2025 18:34

CUMULUS-3960: Updated PostToCmr task to be able to republish granules

acd395b

upgrade localstack to fix aws-sdk error

4f790ff

jennyhliu added the work in progress label Jan 21, 2025

jennyhliu added 6 commits January 21, 2025 19:37

fix code

e9e1831

localstack return undefined for sqs.receiveMessage

548fbf1

unpublish granule without cmrLink

e356b99

Revert "upgrade localstack to fix aws-sdk error"

863fe9b

This reverts commit 4f790ff.

Revert "localstack return undefined for sqs.receiveMessage"

6050fde

This reverts commit 548fbf1.

aggregateError add configuration

c7d9b94

jennyhliu commented Jan 24, 2025

View reviewed changes

jennyhliu added 2 commits January 24, 2025 19:33

fix pass in parameter

13fa33f

stopOnError

d255ece

jennyhliu commented Jan 27, 2025

View reviewed changes

jennyhliu and others added 4 commits January 27, 2025 16:01

Merge branch 'master' into jl/CUMULUS-3960

08128fc

unit test

ef1a31e

remove integrationworkflow config add to unit test

384b3af

only one line not covered

7de83d4

jennyhliu commented Jan 28, 2025

View reviewed changes

update readme

958e18a

jennyhliu marked this pull request as ready for review January 29, 2025 15:10

Merge branch 'master' into jl/CUMULUS-3960

ad40a0f

jennyhliu added needs review and removed work in progress labels Jan 29, 2025

jennyhliu added 2 commits January 29, 2025 12:06

correct type

a9072ef

Merge branch 'jl/CUMULUS-3960' of https://github.com/nasa/cumulus int…

cf2103b

…o jl/CUMULUS-3960

Nnaga1 reviewed Jan 30, 2025

View reviewed changes

Nnaga1 added in review and removed needs review labels Jan 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

jennyhliu commented Jan 21, 2025 •

edited

Loading

jennyhliu Jan 24, 2025

jennyhliu Jan 27, 2025

Nnaga1 Jan 30, 2025

jennyhliu Jan 28, 2025 •

edited

Loading

Nnaga1 left a comment •

edited

Loading

Nnaga1 Jan 30, 2025

Nnaga1 Jan 30, 2025

Nnaga1 Jan 30, 2025

Nnaga1 Jan 30, 2025

jennyhliu Jan 31, 2025

Nnaga1 Jan 30, 2025

jennyhliu Jan 31, 2025

Nnaga1 Jan 30, 2025

Nnaga1 Jan 30, 2025

Nnaga1 Jan 30, 2025

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

Are you sure you want to change the base?

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

Conversation

jennyhliu commented Jan 21, 2025 • edited Loading

Changes

Testing

PR Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennyhliu Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Nnaga1 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennyhliu commented Jan 21, 2025 •

edited

Loading

jennyhliu Jan 28, 2025 •

edited

Loading

Nnaga1 left a comment •

edited

Loading