Ecarton/cumulus 3751 from 18.5.2 #3900

etcart · 2025-01-10T16:24:30Z

Summary: task which takes granules and a target collection, updating granules to belong to that target collection in s3 and cumulus datastore, idempotently

Addresses CUMULUS-3751: Workflow task that updates a granule to a new collection

Changes

adds new task to update S3 and cumulus data stores moving granules across collections
adds integration tests and associated resources in the example project
re-distributes ecs cluster resources among ecs tasks in example project

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

* Fix isThrottlingException function to check error name * update changelog and add name/code check in errors * linter fix * changelog * typo fix --------- Co-authored-by: Hailiang Zhang <[email protected]> Co-authored-by: etcart <[email protected]>

…s_table (#3805) Co-authored-by: etcart <[email protected]>

* Update deployment templates for Aurora Serverless v2 (#3623) * update CL * update terraform templates to serverless v2 * add terraform variable validation * remove upgrade variables * add prevent_destroy = true * add prevent_destroy = true * CUMULUS-3670 Develop upgrade/migration process Aurora Serverless v1 to v2 (#3643) * remove prevent_destroy to allow automated CI migrations * set force_ssl = 0 (#3658) Co-authored-by: Tim Clark <[email protected]> * [CUMULUS-3671]: Update docs for Serverless V2 (#3666) * initial commit * serverless v2 doc updates * Update serverless V2 docs * Fix lint issue * set DISABLE_PG_SSL: true to support CI * fix lint error * set disableSSL = true * remove DISABLE_PG_SSL * set rejectUnauthorized: 'false' * update CL for v2 changes * fix changelog * add migration notes to changelog, add v2 docs to sidebar * fix changelog --------- Co-authored-by: Tim Clark <[email protected]> Co-authored-by: Nate Pauzenga <[email protected]>

) * Update AWS errors to use the V3 error classes * Fix lint * Import aws sdk directly to avoid circular dep * Update CL * Remove module in favor of aws imports directly * Revert change to ThrottlingException error type * Add comments * Fix lint * Remove unnecessary dependency * add debug logging for CI * update type and debug comment * temporarily revert to name checks * Remove logging and type check on conditional exception. Instance of does not work in this case. I believe we're calling the service "dynamodbDocClient" using the non-V3 syntax. * Fix lint * Update tests to throw correct aws-sdk error * Update tests with new aws-sdk error types * Import error type correctly * Correctly import sfn error * Instantiate errors like I know what I'm doing * Basic syntax 🤦 * update tests * Remove unnecessary comment * fixup for clarity * Update test for clarity * Update test fixture and logging for consistency

* Allow override of sfEventSqsLambda timeout with associated queue adjustments * Update CHANGELOG * Respond to PR feedback * Update per PR feedback request

* CUMULUS-3906 - Update to ORCA v10.0.0 * Resolved CL conflict. * Removes required wording for 3906 from CL

* Fixes merge conflict * Adds diff link for v18.5.0

…e-granules-cmr-metadata-… (#3791) * Added excludeFileRegex configuration to update-granules-cmr-metadata-file-links (#3790) Updated tests to exercise new file-exclusion feature * linter fixes * remove explicit null for un-found regexpattern * switch to logging when no excludable files found * changelog broken into multiple lines * linter fixes in changelog * name in changelog after lambda function name * remove TODO. non-mocked is a truer representation of function * small refactor * typo in passthrough of fileregex * nyc values with new tests * version requirement update * fixed merge weirdness * fix jsonpath in the other places it's flagged * remove unneeded explicit pin in aws-client * check like instead of deepequal on credentials return --------- Co-authored-by: Mike Dorfman <[email protected]>

* Fix cumulus versions due to bad merge * Update aws-sdk versions to revert bad merge

* update dependencies to latest cma, cma-python, cumulus-process * changelog * fixed shas in locks * whitelist jsonpath for buiggy audit behavior * remove incorrect changelog entry --------- Co-authored-by: etcart <[email protected]>

* CUMULUS-3891: Add fastGet download option to sftp data file download * add sftpFastDownload config * fix fastDownload boolean vs string * add unit tests * fix aws-client services unit test * test SFTP_DEBUG * remove only * add changelog entry * remove unused code * remove jsonpath from common * update latest-version and add jsonpath-plus to audit-ci * serial * update readme remove serial * add sftp test

My first PR :-)

…3830) * Update Orca version * Update orca variables for v10 release series * Update orca var to default value * Update Orca version to official release

…ync-granules (#3823) * Iniital commit updating sync-granule behavior * Clean up comments * Update schema config to match changes * Update typings * Fix sync-granules typing * Update config docs to true * Update task README * Update CHANGELOG * Minor fix * Update spec tests with new default hashed granId path * Update @cumulus/types to allow for explicit export of api/collections * Abstract typings to seperate file * Update _ingestGranule param based on PR feedback * Fix unit test not updated on merge * Add method unit tests for collection(name/version)From methods * Fix integration helper * Add hashed path to SyncGranules

…zed (#3832) * CUMULUS-3919:Added terraform variables disableSsl and rejectUnauthorized * disableSsl->disableSSL

…s services. (#3838) * Allowing force_new_deployment to be configurable for ecs services. * Update CHANGELOG --------- Co-authored-by: Michael Hall <[email protected]>

* guarantee non-numeric nonNumericString * bringing in pg8.13 snyk suggestion * trying to kcik up the sync-granules task error * trying to get repeatable tries to sync-granules * focus on just the important part in syncgranule * trying to get publish to work * does only this need to change to import? * move to import function import * reverting bad code for testing reasons * changelog * keep string length the same, no reason to twiddle this

* Release 19.1.0 (#3816) * version bump * Update CL * Update docs * Update CL link * Add note for clarity * update missed deps * Ecarton/cumulus 3928 imf work (#3831) (#3841) * guarantee non-numeric nonNumericString * bringing in pg8.13 snyk suggestion * trying to kcik up the sync-granules task error * trying to get repeatable tries to sync-granules * focus on just the important part in syncgranule * trying to get publish to work * does only this need to change to import? * move to import function import * reverting bad code for testing reasons * changelog * keep string length the same, no reason to twiddle this Co-authored-by: etcart <[email protected]> * fix no top level await (#3843) (#3844) Co-authored-by: etcart <[email protected]> --------- Co-authored-by: etcart <[email protected]>

* Update CHANGELOG from release 18.5.1 * Add CHANGELOG footer * Fix CHANGELOG

* Passed through sqs_message_consumer_watcher_time_limit and sqs_message_consumer_watcher_message_limit * Added a changelog entry for CUMULUS-3904 --------- Co-authored-by: mikedorfman <[email protected]>

* CUMULUS-3876:Fix S3 Replicator cross region bucket writes * add target_region variable * fix typing * add unit test * add target_region to deployment * covert to ts * add s3-replicator.tf * update tsconfig and webconfig * remove s3-replicator example * update package.json * update changelog * fix types * add s3-replicator.tf.example * update typo

* Fix 2564 docs * Update sync-granules doc to be correct

jennyhliu · 2025-01-17T21:47:00Z

example/spec/parallel/moveGranuleCollections/MoveGranuleCollectionWorkflowSpec.js

+
+const activityStep = new ActivityStep();
+
+describe('The MoveGranuleCollection workflow using ECS', () => {


What are the source and target collections? I can't tell from the specs.

What are the difference between MoveGranuleCollectionWorkflowSpec and MoveGranuleCollectionsSpec?

The intent is to run movegranulecollectionsSoec as a test of the functionality of the lambda, while the workflow is meant to run the workflow as it would exist with example data, to be added to with CMR, lizards etc when those tickets are done. Right now the only functional difference is that it tests the lambda in the ECS deployment

jennyhliu · 2025-01-17T21:54:27Z

example/spec/parallel/moveGranuleCollections/move-granule-collection-spec-utils.js

+
+const getSourceCollection = (sourceUrlPrefix) => (
+  {
+    files: [


This should be part of the collection configuration, and shouldn't need to specify here. We ingest different granules/files to s3 to avoid conflicts.

jennyhliu · 2025-01-17T21:57:03Z

example/spec/parallel/moveGranuleCollections/move-granule-collection-spec-utils.js

+  }
+);
+
+const getTargetCollection = (targetUrlPrefix) => ({


Same comment: This should be part of the target collection configuration

packages/api-client/src/granules.ts

packages/cmrjs/src/types.ts

jennyhliu · 2025-01-17T22:18:51Z

tasks/move-granule-collections/README.md

+| ---------- | ---- | ------- | ------ | -----------
+
+| buckets | object | (required) | | Object specifying AWS S3 buckets used by this task
+| collection | object | (required) | | The cumulus-api collection object


This is 'target' collection, right? Should this be part of the task input? Since the task can be invoked to move granules to different collections.

tasks/move-granule-collections/src/index.ts

jennyhliu · 2025-01-17T22:37:53Z

tasks/move-granule-collections/src/index.ts

+  };
+}
+
+async function buildTargetGranules(


So buildTargetGranules does not only build but also update granule metadata in s3?
Where is the metadata got removed?
I'm confused with the sequence of events.

This builds the granule records as they should exist once we're done, so that they can later be used as a roadmap for the updates in s3 and pg

Co-authored-by: jennyhliu <[email protected]>

etcart · 2025-01-20T16:13:22Z

I'm thinking of pulling the example ecs deployment here. it was put in place to allow for exceeding 15 minutes (and other potential resource constraints). but splitting into subsets of granule_ids should allow us to control for that, an date ecs example adds (unnecessary?) complexity to this pr

…e collection

… to 3.726 due to unit failure in > 3.729.0 (#3905) * Pin @aws-sdk/client-s3 to 3.726 due to unit failure in > 3.729.0 * Fixup * Fix dependency concern * Update CHANGELOG

jennyhliu · 2025-01-23T16:17:37Z

tasks/move-granule-collections/.gitignore

I don't see other tasks/* have this file

removed here
remove gitignore from task

jennyhliu · 2025-01-24T17:39:00Z

example/cumulus-tf/ecs_move_granule_collections.asl.json

How are we going to avoid the start of the workflow triggering the granule updates, since the input payload has granules?

would a simple. rename to "granuleIds" solve this?

…ulus-3751-from-18.5.2

cumulus-bot and others added 30 commits September 19, 2024 14:02

Merge branch 'master' of https://github.com/nasa/cumulus

ef1dacc

reintroduce Migration Count Report to migrations with async_operation…

0f0cd98

…s_table (#3805) Co-authored-by: etcart <[email protected]>

Merge branch 'master' of https://github.com/nasa/cumulus

67b6145

Jk/cumulus 3020 (#3800)

3192e08

* Allow override of sfEventSqsLambda timeout with associated queue adjustments * Update CHANGELOG * Respond to PR feedback * Update per PR feedback request

CUMULUS-3906 - Update to ORCA v10.0.0 (#3813)

2b25cfa

* CUMULUS-3906 - Update to ORCA v10.0.0 * Resolved CL conflict. * Removes required wording for 3906 from CL

CUMULUS-3897 - Merges v18.5.0 CL updates (#3817)

82abf69

* Fixes merge conflict * Adds diff link for v18.5.0

CUMULUS-3897 - Adds release 18.5.0 website docs (#3819)

151ec9c

Reverts ORCA v10.0.0 update (#3821)

5342484

Fix cumulus versions due to bad merge (#3826)

edb9f7b

* Fix cumulus versions due to bad merge * Update aws-sdk versions to revert bad merge

Just adding in my creds to the CI. (#3827)

0f0b794

My first PR :-)

[CUMULUS-3921] Test new Orca version and update example deployment (#…

199cdcd

…3830) * Update Orca version * Update orca variables for v10 release series * Update orca var to default value * Update Orca version to official release

CUMULUS-3919:Added terraform variables disableSSL and rejectUnauthori…

02b42fb

…zed (#3832) * CUMULUS-3919:Added terraform variables disableSsl and rejectUnauthorized * disableSsl->disableSSL

Minor document typo/fix (#3834)

c16fe6f

CUMULUS-3931: Allowing force_new_deployment to be configurable for ec…

d1e0311

…s services. (#3838) * Allowing force_new_deployment to be configurable for ecs services. * Update CHANGELOG --------- Co-authored-by: Michael Hall <[email protected]>

fix no top level await (#3843)

d340f1e

Merge branch 'master' of https://github.com/nasa/cumulus

adac427

Update CHANGELOG from release 18.5.1 (#3845)

9b51d05

* Update CHANGELOG from release 18.5.1 * Add CHANGELOG footer * Fix CHANGELOG

Cumulus 3904 (#3837)

2fe02e6

* Passed through sqs_message_consumer_watcher_time_limit and sqs_message_consumer_watcher_message_limit * Added a changelog entry for CUMULUS-3904 --------- Co-authored-by: mikedorfman <[email protected]>

CUMULUS-2564 - Fix 2564 docs (#3856)

fcfe2db

* Fix 2564 docs * Update sync-granules doc to be correct

Update bootstrap/pdr-upsert tests to run serially (#3855)

7730d99

jennyhliu reviewed Jan 17, 2025

View reviewed changes

etcart and others added 3 commits January 17, 2025 19:08

Update CHANGELOG.md

392aa85

Co-authored-by: jennyhliu <[email protected]>

WIP still figuring out the right way to update the cmr file

06f5f68

Merge branch 'ecarton_rebase-3751' into ecarton/cumulus-3751-from-18.5.2

8ce2e2e

etcart and others added 13 commits January 21, 2025 11:31

move to using api to get granule data from ID

db8c1da

shift tests to using granuleId as input

6c9fa7a

test files have granule id payload instead of granule payload

cae4527

sourceCollection in code and tests to allow update with details befor…

ab0f609

…e collection

small format fix

0134166

config and linter errors

4a59394

more linting

65f4673

fix input configuration

148a291

double check my inputs

21fe9f9

let me make sure the config is right

dacde25

CUMULUS-3967 -- Forward port of release fix -- Pin @aws-sdk/client-s3…

762b655

… to 3.726 due to unit failure in > 3.729.0 (#3905) * Pin @aws-sdk/client-s3 to 3.726 due to unit failure in > 3.729.0 * Fixup * Fix dependency concern * Update CHANGELOG

think this needs to be corected

d46049b

remove unnecessary eslints

707e101

jennyhliu reviewed Jan 23, 2025

View reviewed changes

etcart added 3 commits January 23, 2025 19:32

moving to non-hardcoded buckets

6bd075e

CL for memory reservation

5180b4a

remove gitignore from task

534a951

jennyhliu reviewed Jan 24, 2025

View reviewed changes

etcart added 4 commits January 26, 2025 12:41

appears to be workign bu tstill need to test cmr udates

49295be

remove unneeded import o CMRFile

456df87

Merge remote-tracking branch 'origin/release-18.5.x' into ecarton/cum…

f7965e9

…ulus-3751-from-18.5.2

accidentally commited mangled ts output

220ea3c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ecarton/cumulus 3751 from 18.5.2 #3900

Ecarton/cumulus 3751 from 18.5.2 #3900

etcart commented Jan 10, 2025 •

edited

Loading

jennyhliu Jan 17, 2025 •

edited

Loading

jennyhliu Jan 17, 2025

etcart Jan 18, 2025

jennyhliu Jan 17, 2025 •

edited

Loading

jennyhliu Jan 17, 2025

jennyhliu Jan 17, 2025

jennyhliu Jan 17, 2025

etcart Jan 18, 2025

etcart commented Jan 20, 2025

jennyhliu Jan 23, 2025

etcart Jan 24, 2025

jennyhliu Jan 24, 2025

etcart Jan 26, 2025


		const activityStep = new ActivityStep();

		describe('The MoveGranuleCollection workflow using ECS', () => {

Ecarton/cumulus 3751 from 18.5.2 #3900

Are you sure you want to change the base?

Ecarton/cumulus 3751 from 18.5.2 #3900

Conversation

etcart commented Jan 10, 2025 • edited Loading

Changes

PR Checklist

jennyhliu Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennyhliu Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etcart commented Jan 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etcart commented Jan 10, 2025 •

edited

Loading

jennyhliu Jan 17, 2025 •

edited

Loading

jennyhliu Jan 17, 2025 •

edited

Loading