Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 #3309

Open
wants to merge 24 commits into
base: develop
Choose a base branch
from
Open

Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 #3309

wants to merge 24 commits into from

Conversation

CoryMartin-NOAA
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA commented Feb 7, 2025 •
edited
Loading

Description

This PR enables the ability to cycle C1152 atm+aerosol on WCOSS2 to support GFSv17.
Wallclock times were extended because 1) model execution is now slower with GOCART (this should probably be configurable in the config.resources depending on APP but that is outside the scope of my PR) 2) the aerosol DA variational solver needs more time at this resolution and 3) it takes FOREVER to copy C1152 restart files with the GOCART aerosol species in them.

Also, the prep job runs out of memory so this spreads it out over more nodes. There are also misc. things that were added to get things to run on WCOSS2 that may need altered/removed post the most recent RFC.

Finally, GDAS hash is updated.

Resolves #3275

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES
    • EMC verif-global
    • GDAS already merged, updating to develop here
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

  • Clone and build on WCOSS
  • Cycled test on WCOSS

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

CatherineThomas-NOAA and others added 15 commits December 19, 2024 09:31
@CatherineThomas-NOAA
Updates for cycling C1152 ATM
87db086 
A handful of updates are needed to cycle with C1152
atmosphere, which are mostly related to configs and
resources.

Refs: NOAA-EMC#3173
@CatherineThomas-NOAA
Merge branch 'NOAA-EMC:develop' into c1152_cyc
7c3a155
@CatherineThomas-NOAA
Merge branch 'NOAA-EMC:develop' into c1152_cyc
9c1b856
@CatherineThomas-NOAA
Update gsi_utils hash for C1152 cycling
c7b3010 
The calc_analysis utility was updated to handle the
increased horizontal resolution of C1152 ATM.

Refs: NOAA-EMC#3173
@CatherineThomas-NOAA
Update gsi_utils for C1152 cycling
a161bf2 
The original calc_analysis bugfix was not suitable for
JEDI-based ATM DA.  This hash update allows for
calc_analysis to work with both GSI and JEDI-based ATM DA.

Resolves NOAA-EMC#3173
@CatherineThomas-NOAA
Merge branch 'NOAA-EMC:develop' into c1152_cyc
e90db09
aerosol c1152 cycling changes
a11f4d2
more resource changes
d9c1c5b
extend walltime for aerovar
6db13da
Merge branch 'develop' into feature/aero1152
27dcbbb
@CoryMartin-NOAA
Merge branch 'NOAA-EMC:develop' into feature/aero1152
c29244f
@CoryMartin-NOAA
update config.aeroanlgenb
c9ef39b
@CoryMartin-NOAA
Merge branch 'NOAA-EMC:develop' into feature/aero1152
c0f96d1
@CoryMartin-NOAA
Merge branch 'NOAA-EMC:develop' into feature/aero1152
4e48fa2
@CoryMartin-NOAA
update GDAS hash
dd9823c
@CoryMartin-NOAA CoryMartin-NOAA self-assigned this Feb 7, 2025
aerorahul
aerorahul reviewed Feb 7, 2025
View reviewed changes
DavidHuber-NOAA
DavidHuber-NOAA requested changes Feb 7, 2025
View reviewed changes
@RussTreadon-NOAA
Copy link
Contributor

@CoryMartin-NOAA , when was the last time you ran build_gdas.sh on WCOSS2 for this PR?

Following RFC 13644 I found it necessary to add the line

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"

to sorc/gdas.cd/build.sh on WCOSS2.

I did so in a working copy of build.sh as

@@ -87,6 +87,10 @@ case ${BUILD_TARGET} in
     ;;
 esac
 
+if [[ $BUILD_TARGET == "wcoss2" ]]; then
+  export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"
+fi
+
 CMAKE_OPTS+=" -DCLONE_JCSDADATA=$CLONE_JCSDADATA -DMACHINE=$BUILD_TARGET"

@RussTreadon-NOAA
Copy link
Contributor

Sorry, it appears g-w practice is to only assign issues to milestones.

@RussTreadon-NOAA RussTreadon-NOAA removed this from the GFS v17 milestone Feb 8, 2025
DavidHuber-NOAA
DavidHuber-NOAA previously approved these changes Feb 13, 2025
View reviewed changes
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks!

aerorahul
aerorahul previously approved these changes Feb 14, 2025
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

RussTreadon-NOAA
RussTreadon-NOAA reviewed Feb 17, 2025
View reviewed changes
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments are questions and not a request for changes. I was unware that we can run gdas.x with multiple threads. Just asking for confirmation that this is indeed true.

I did not run any g-w CI on any machines for this PR. I can apply CI labels if the g-w team wants. Just let me know.

DavidHuber-NOAA
DavidHuber-NOAA approved these changes Feb 18, 2025
View reviewed changes
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me!

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Feb 20, 2025
@emcbot emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Feb 20, 2025
@emcbot
Copy link

emcbot commented Feb 20, 2025

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3309/RUNTESTS on WCOSS

@KateFriedman-NOAA
Copy link
Member

Restarting CI on WCOSS2 (no fault of this PR).

@KateFriedman-NOAA KateFriedman-NOAA removed the CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress label Feb 20, 2025
@emcbot emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Feb 20, 2025
@emcbot
Copy link

emcbot commented Feb 20, 2025

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3309/RUNTESTS on WCOSS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RussTreadon-NOAA RussTreadon-NOAA RussTreadon-NOAA left review comments

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA approved these changes

@aerorahul aerorahul aerorahul left review comments

@CatherineThomas-NOAA CatherineThomas-NOAA Awaiting requested review from CatherineThomas-NOAA

@TravisElless-NOAA TravisElless-NOAA Awaiting requested review from TravisElless-NOAA

Assignees

@CoryMartin-NOAA CoryMartin-NOAA

Labels
CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress JEDI Feature development to support JEDI-based DA
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update defaults to support C1152 aerosol DA for GFSv17
8 participants
@CoryMartin-NOAA @RussTreadon-NOAA @emcbot @KateFriedman-NOAA @aerorahul @WalterKolczynski-NOAA @DavidHuber-NOAA @CatherineThomas-NOAA