Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues/23: Enable Harmony write output to user bucket #46

Merged
merged 16 commits into from
Dec 17, 2024
Merged

Issues/23: Enable Harmony write output to user bucket #46

merged 16 commits into from
Dec 17, 2024

Conversation

frankinspace
Copy link
Member

Github Issue:

#23 Harmony requests now include destinationUrl parameter to place output directly in s3 bucket instead of requiring data to be copied.

Also addressed

#9 Added some documentation for installing as cumulus module
#41 Module no longer depends on deprecated hashicorp/template provider
#42 Terraform version upgraded to v1.5.3

Description

Module will now allow Harmony to write output to a staging bucket directly instead of copying data from Harmony.

Overview of work done

  • Added new module variable bignbit_staging_bucket. If left blank a new S3 bucket will be created with permissions for Harmony to write output to the bucket and GIBS to read from the bucket.
  • Replaced copy to s3 lambda function with new process harmony results lambda function
  • Move lambda role creation to within the module instead of needing to be declared externally and passed as variable.
  • Updated readme with better installation instructions
  • Removed deprecated hashicorp/template provider
  • Upgraded terraform to v1.5.3

Overview of verification done

Deployed to SIT and successfully executed both OPERA HLS and S1 granules

Overview of integration done

Will test further in UAT with GIBS.

PR checklist:

  • Linted
  • Updated unit tests
  • Updated changelog
  • Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

@frankinspace frankinspace requested a review from torimcd December 16, 2024 20:51
This was linked to issues Dec 16, 2024
Update harmony step to place results directly in s3 #23
Closed
Update documentation to explain how to install cumulus module #9
Closed
Migrate away from depreciated hashicorp/template #41
Closed
Upgrade terraform version #42
Closed
@frankinspace frankinspace merged commit 8e11bc6 into develop Dec 17, 2024
4 of 5 checks passed
@frankinspace frankinspace deleted the issues/23 branch December 17, 2024 15:08
frankinspace added a commit that referenced this pull request Feb 4, 2025
@frankinspace @voxparcxls
@torimcd @jake-herrmann @jamesfwood
Merge pull request #49 from podaac/release/0.2.0
43d56cb 
* /version 0.2.0a0

* feature: convertToPNG --> sendToHarmony (#21)

* update state_machine, sendToHarmony

* --changelog Issue-15

* moved to unreleased, changelog

* /version 0.2.0a1

* Update README.md (#22)

* /version 0.2.0a2

* Use -j to remove top-level directory in zip artifact (#25)

* /version 0.2.0a3

* Issue#16: Change apply opera treatment choice and lambda to be specific to HLS (#24)

* update for HLS -- lambda and state machine

* py lint error

* update file name

* /version 0.2.0a4

* /version 0.2.0a5

* Feature/issue 7 - Remove wait for GITC response (#30)

* fix type in readme

* change task token to uuid in send to gitc

* remove wait for task token & gitc response handler from state machine

* change gitc response handler to invoke save cma message

* fix typo

* fix context parameter

* convert uuid to string

* fix json typo

* remove uuid lib to use stdlib uuid

* change identifier to image set name instead of uuid

* remove uuid import

* fix json formatting of cma invoke

* add granule concept ID to identifier

* update image set name with granule conceptid

* remove whitespace

* move get umm json to utils and save gitc outgoing cnm

* remove unused import

* try setting cmr query env based on stage

* remove task token from tests

* fix granule index

* change how image set name defined

* add region to ssm client in utils

* add ssm parameters to gitc lambdas

* change region reference

* update how granule name referenced in send to gitc

* remove unused cmr var

* fix case

* fix cnm parsing

* reformat save cnm to separate step

* update tf vars for save cnm

* fix type in module definition

* fix cnm in cma

* fix collection reference in save cnm

* add debugging log statements

* fix input

* update save cnm input

* debugging

* debugging

* linting

* debugging

* debugging

* linting

* fix state machine

* update state machine

* reorg state machine

* missing comma

* move save cnm into map

* fix state machine transitions

* fix sm

* fix boolean

* debugging cnm input

* fix input

* change input

* debug input

* add cnm as config parameter

* lint

* fix cnm payload

* remove debugging statements

* add prefix to cnm path and fix gitc response

* increase handle gitc response timeout

* update changelog

* add original shortname as cnm prefix

* revert shortname and change collection ref to save cnm & cnm-r in same location

* remove save cma lambda no longer used

* remove EDL env params from sendtogitc

* change parsing of granule concept id

* /version 0.2.0a6

* Update README.md

* /version 0.2.0a7

* Support datetimes without microseconds (#37)

* Support datetimes without microseconds

* changelog

* Update generate_image_metadata.py (#39)

add docstring

* /version 0.2.0a8

* /version 0.2.0a9

* Issues/23: Enable Harmony write output to user bucket (#46)

* add s3 destination to harmony job

* syntax

* job success

* harmony block update

* --variable

* temp test in uat

* manual to uat

* test s1-v1

* lint

* deploy --sit

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

* Update to place harmony results in our bucket

---------

Co-authored-by: Josh <[email protected]>

* /version 0.2.0a10

* fix deploy failure

* /version 0.2.0a11

* Add new message attribute response_topic_arn to all outgoing messages to GIBS (#47)

* /version 0.2.0a12

* prep for 0.2.0

* /version 0.2.0a13

* Fix build error

* /version 0.2.0a14

* /release 0.2.0

* Update graph

* /version 0.2.0rc1

* Turn prevent_destroy back on for gibs response topic

* /version 0.2.0rc2

* Improve error message when missing config

* /version 0.2.0rc3

* Improve error message when missing config

* /version 0.2.0rc4

* UAT is in east-1 region

* /version 0.2.0rc5

* fix gibs queue name

* /version 0.2.0rc6

* calculate md5sum for files from harmony

* calculate md5sum for files from harmony

* /version 0.2.0rc7

* bump chunk size to 100 MB

* /version 0.2.0rc8

* Update test message for UAT, add checksum to generated tif, add logging, fix type error

* Lint

* Lint

* /version 0.2.0rc9

* Change named variables for audit path from pobit to bignbit

* /version 0.2.0rc10

* Update docs

---------

Co-authored-by: frankinspace <[email protected]>
Co-authored-by: Josh Haile <[email protected]>
Co-authored-by: Victoria McDonald <[email protected]>
Co-authored-by: Jake Herrmann <[email protected]>
Co-authored-by: James Wood <[email protected]>
Co-authored-by: jamesfwood <[email protected]>
Co-authored-by: Josh <[email protected]>
@frankinspace frankinspace mentioned this pull request Mar 12, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@torimcd torimcd torimcd approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@frankinspace @torimcd @voxparcxls