Cumulus v18.3.1 Upgrade

[gjclark]

No description provided.

[mattp0]

Curious what people think we should do here in core:

CUMULUS-3433 Update to node.js v20
The following applies only to users with a custom value configured for
async_operation_image:

As part of the node v20 update process, a new version (49) of the Core
async-operation container was published - cumuluss/async
operation The
default value for async_operation_image has been updated in the cumulus
module, however if you are using an internal image repository such as ECR,
please make sure to update your deployment configuration with the newly
provided image.

Users making use of a custom image configuration should note the base image
for Core async operations must support node v20.x.

CUMULUS-3617 Migration of DLA messages should be performed after Cumulus is upgraded
Instructions for migrating old DLA (Dead Letter Archive) messages to new format:

YYYY-MM-DD subfolders to organize by date
new top level fields for simplified search and analysis
captured error message
To invoke the Lambda and start the DLA migration, you can use the AWS Console or CLI:

aws lambda invoke --function-name $PREFIX-migrationHelperAsyncOperation
--payload $(echo '{"operationType": "DLA Migration"}' | base64) $OUTFILE
PREFIX is your Cumulus deployment prefix.
OUTFILE (optional) is the filepath where the Lambda output will be saved.
The Lambda will trigger an Async Operation and return an id such as:

{"id":"41c9fbbf-a031-4dd8-91cc-8ec2d8b5e31a","description":"Migrate Dead Letter Archive Messages",
"operationType":"DLA Migration","status":"RUNNING",
"taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789"}
which you can then query the Async Operations API
Endpoint for the
output or status of your request. If you want to directly observe the progress
of the migration as it runs, you can view the CloudWatch logs for your async
operations (e.g. PREFIX-AsyncOperationEcsLogs).

CUMULUS-3779 async_operations Docker image version upgrade
The async-operation Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least async-operations:52.

CUMULUS-3776 cumulus-ecs-task Docker image version upgrade
The cumulus-ecs-task Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least cumulus-ecs-task:2.1.0.

there appears to be nothing to be done in core for this. The lambda is an invoke with this payload. IMO its seems like it wont need a migration script.

The container updates are more an FYI..

[mattp0]

Some context for why we are moving to Amazon Linux 2023. Node 20.x can not be installed on Amazon Linux 2 because the node system required a GLIBC 2.28 and Amazon Linux 2 does not have a non trivial way of doing this. We found many blog posts of people asking if GLIBC 2.27+ is possible on Amazon Linux 2 with no solutions.

What this means is the baseline python version is now 3.9. ASF is prepared to accept this and handle updating our dependencies. We would like feedback from other DAACs here if this is reasonable.

There is a chance we could install 3.8 on the container but if no one has a strong reason to support 3.8 I would be in favor of moving forward from 3.8 as its EOL in Oct 2024.

What do you all think? @lindsleycj @mikedorfman

[lindsleycj]

@mattp0

There is a chance we could install 3.8 on the container but if no one has a strong reason to support 3.8 I would be in favor of moving forward from 3.8 as its EOL in Oct 2024.

What do you all think? @lindsleycj @mikedorfman

I am fine with removing python 3.8 from this.

[lindsleycj]

I'm fine with these changes.

[mikedorfman]

Some context for why we are moving to Amazon Linux 2023. Node 20.x can not be installed on Amazon Linux 2 because the node system required a GLIBC 2.28 and Amazon Linux 2 does not have a non trivial way of doing this. We found many blog posts of people asking if GLIBC 2.27+ is possible on Amazon Linux 2 with no solutions.

What this means is the baseline python version is now 3.9. ASF is prepared to accept this and handle updating our dependencies. We would like feedback from other DAACs here if this is reasonable.

There is a chance we could install 3.8 on the container but if no one has a strong reason to support 3.8 I would be in favor of moving forward from 3.8 as its EOL in Oct 2024.

What do you all think? @lindsleycj @mikedorfman

I am fine with this as well. Thanks!