Mpas testing #2
Open
Mpas testing #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n 2-meter temperature and 2-meter water vapor mixing ratio to the output_noahmp diagnostic pool. The variables t2mxy and q2mxy are computed in Noah-MP LSM as functions of the vegetation fraction, and the 2-meter temperatures and water vapor mixing ratios over vegetation and bare soil (t2mvxy and t2mbxy; q2mvxy and q2mbxy), but were not directly output as diagnostics.
…fied the call to
subroutine seaice_noah.
-> in ./physics_wrf/module_sf_noah_seaice_drv.F, subroutine seaice_noah includes the
optional variables noahres, potevp, and snopcx. these three variables are used in
the Noah LSM but not used in the Noah-MP LSM. separate calls to seaice_noah as a
function of the LSM option reflects this difference between Noah and Noah-MP in
subroutine driver_seaice.
-> in mpas_atmphys_driver.F, modified calls to allocate_seaice and deallocate_seaice
accordingly.
…t2m, and th2m when
running the Noah-MP land surface scheme:
-> added the module mpas_atmphys_sfc_diagnostics.F to compute q2,t2m,and th2m for the
Noah and Noah-MP land surface schemes, separately.
-> removed the computation of q2,t2m,and th2m for the Noah LSM in mpas_atmphys_driver_seaice.F.
-> in mpas_atmphys_driver.F, added the call to atmphys_sfc_diagnostics in subroutine
physics_driver.
-> corrected Makefile accordingly.
…ed the initialization of variable sh2o (soil liquid water) prior to calling subroutine NoahmpInitMain in subroutine noahmp_init.
…the dimension of index_nifa and index_nwfa from the pool lbc_state instead of state. Corrected the call to subroutine init_atm_thompson_aerosols_lbc accordingly. * In ./src/core_init_atmosphere/mpas_init_atm_thompson_aerosols.F, replaced the index name of nifa and nwfa from index_nifa (index_nwfa) to index_lbc_nifa (index_lbc_nwfa) from the pool lbc_state.
…2.3 (PR MPAS-Dev#1249) This merge corrects the pool from which lbc_scalar constituent indices are obtained in the init_atm_thompson_aerosols_lbc routine. Rather than obtaining index_nifa and index_nwfa from the state pool, the indices of lbc_nifa and lbc_nwfa should be obtained from the lbc_state pool. * NSF-MPAS/correct_thompson_aerosols_lbc: * In ./src/core_init_atmosphere/mpas_init_atm_thompson_aerosols, get the...
…ev#1244) This merge corrects the computation of the soil temperature (TSLB) in the Noah-MP land surface scheme through the addition of initialization of the soil liquid water (SH2O) in the noahmp_init subroutine in module mpas_atmphys_lsm_noahmpinit.F prior to calling NoahmpInitMain. Prior to the changes in this merge, running Noah-MP led TSLB to be set to ConstFreezePoint (273.16 K) in SoilSnowWaterPhaseChangeMod.F90 although the input TSLB was greater than ConstFreezePoint on the very first time step. This error occurred because TSLB is calculated as a function of the soil liquid (MassWatLiqTmp) and ice (MassWatIceTmp) water mass, and MassWatLiqTmp is always equal to 0 when not properly initializing SH2O. * NSF-MPAS/noahmp_fix_tslb: * In ./src/core_atmosphere/physics/mpas_atmphys_lsm_noahmpinit.F, added the...
-> in Registry.xml, corrected the units of static fields greenfrac, shdmin, shdmax, vegfra,
and albedo12m from "unitless" to "percent".
-> in mpas_init_atm_cases.F, corrected spelling of "greeness" to "greenness" (line 6702).
* In ./src/core_atmosphere/Registry.xml, corrected the units of static fields greenfrac,
shdmin, shdmax, vegfra, and albedo12m from "unitless" to "percent". corrected spelling of
greenness fraction in the description of greenfrac.
…3 (PR MPAS-Dev#1248) This merge corrects the units of the fields 'greenfrac', 'shdmin', 'shdmax', 'vegfra', and 'albedo12m' from "unitless" to "percent" in the init_atmosphere and atmosphere core Registry.xml files. Also included in this merge is a correction to the spelling of "greenness" in several places. * NSF-MPAS/correct_units_static_fields: * In ./src/core_init_atmosphere: -> in Registry.xml, corrected the units of static fields greenfrac, shdmin, shdmax, vegfra, and albedo12m from "unitless" to "percent". -> in mpas_init_atm_cases.F, corrected spelling of "greeness" to "greenness" (line 6702).
…ock_fields This commit removes a duplicate allocation of indexToEdgeID % array in the mpas_io_setup_edge_block_fields routine that was the source of a memory leak.
…x-v8.2.3 (PR MPAS-Dev#1258) This merge removes a duplicate allocation of indexToEdgeID % array in the mpas_io_setup_edge_block_fields routine that was the source of a memory leak. * framework/fix_duplicate_indexToEdgeID_alloc: Remove duplicate allocation of indexToEdgeID in mpas_io_setup_edge_block_fields
…on of GWDO statistics This commit introduces code changes in order to improve the memory footprint of the computation of Gravity Wave Drag Orography statistics in the init_atmosphere. The previous approach involved each MPI rank reading in the entirety of the topography and land use tiles, and this may prevent fully-subscribing to all cores in a node. The new approach only reads in one tile at a time, when it encounters a pixel whose data is not already available in a linked list. The new algorithm is able to fully subscribe to all ranks in a node, leading to better parallel performance. The get_box subroutine is modified to call get_tile_from_box_point to check if the current pixel in the box is present in the linked list, and if not, it appends this tile (after reading both topo and land use data) to the head of the list. This commit also changes box, box_landuse, dxm, nx and ny to be local variables, instead of module variables. This provides better readability, along with advantages of thread safety. This commit also removes the extraneous conversion of the cell latitude from radians to degrees and back to radians, prior to the estimation of zonal box size. The result is a numerically more correct code, but it results in marginal differences with the previous approach
…halos The cellLimitField field is allocated in the mpas_block_creator_build_cell_halos routine and is only used in a call therein to mpas_dmpar_get_exch_list. If not deallocated at the end of the mpas_block_creator_build_cell_halos routine, the memory allocated to cellLimitField would be otherwise be leaked, since there is nothing outisde of the mpas_block_creator_build_cell_halos routine that retains a reference to it and deallocates it. This commit fixes this memory leak by deallocating cellLimitField before the mpas_block_creator_build_cell_halos routine returns.
…last matching tile This commit introduces an optimization for the lookup of tiles in subroutine get_tile_from_box_point by passing the most recent successful tile lookup to the next iteration of the search. These changes substantially improves the single-core performance of the compute_gwd_fields subroutine, and also improve the parallel performance of this computation.
Small whitespace changes. Also change implicit loop to an explicit loop to better parallelize. Implicit loops can be ported with 'acc kernels', but we prefer more proscribed 'acc parallel ...' constructs.
These changes enable the GPU execution of the
atm_recover_large_step_variables_work subroutine by adding OpenACC directives.
In order to factor out the time to transfer data between CPU and GPU within
this routine, the new timer 'atm_recover_large_step_variables [ACC_data_xfer]'
has been added to the log file.
Changes include:
- Preparing the routine for porting. Modifying whitespace to make regions clear,
changing implicit loop assignments to be explicit, and fusing some loops.
- Adding OpenACC parallel and loop directives to the do-loops.
- Managing the invariant fields needed for this routine in
mpas_atm_dynamics_{init,finalize} so they are available across timesteps.
- Managing the other fields needed in the routine with OpenACC directives and
using default(present) to ensure data isn't missed. default(present) clauses
cause a run-time error if data isn't present and will help as we fuse data
regions.
…n GPUs. Note this commit adds "atm_bdy_set_scalars_work [ACC_data_xfer]" timers to time the data transfers done in atm_bdy_set_scalars_work, but there is no timer for the actual computation done in atm_bdy_set_scalars_work.
…rk on GPUs. Note this commit adds "atm_zero_gradient_w_bdy_work [ACC_data_xfer]" timers to time the data transfers done in atm_zero_gradient_w_bdy_work, but there is no timer for the actual computation done in atm_zero_gradient_w_bdy_work.
…tmosphere This commit adds three new source files mpas_gsl_oro_data_sm_scale.F mpas_gsl_oro_data_lg_scale.F mpas_init_atm_gwd_gsl.F to the init_atmosphere core, along with modifications to the main core_init_atmosphere Makefile to compile these source files. At present, none of the code in these files is called from the init_atmosphere core.
…re core
This commit defines new static fields, a new package, a new namelist option, and
a new output stream for the GSL GWDO scheme. Also included in this commit are
changes to the init_atm_setup_packages() routine to set the new
'gwd_gsl_stage_out' package if and only if the new namelist option
'config_native_gwd_gsl_static' is true.
When 'config_native_gwd_gsl_static' is set in the namelist, the init_atmosphere
core will write the twenty new static fields (var2d, con, oa{1,2,3,4}, and
ol{1,2,3,4}) for large scale (ls) and small scale (ss) to a new 'ugwp_oro_data'
output stream.
As of this commit, code to actually compute the twenty new static fields is not
being called, and so the fields are expected to be zero everywhere.
When the namelist option config_native_gwd_gsl_static is .true., the init_atm_setup_case() routine now calls thecalc_gsl_oro_data() routine to compute the twenty GSL GWDO static fields.
This commit adds an entry to the Externals.cfg file to obtain the GSL UGWP code from a GitHub repository. Also included in this commit are changes to the main physics Makefile to compile the UGWP code. At present, none of the UGWP code is called.
Also included in this commit is logic for setting up packages related to the GSL UGWP scheme in the atm_setup_packages() routine.
…emes This commit adds a new module, module_bl_ugwp_gwdo, in the core_atmosphere/physics/physics_wrf directory. This module serves as a wrapper for the UGWP physics that is compatible with the MPAS-A physics drivers. Changes to the Makefile in the physics_wrf directory ensure that the module_bl_ugwp_gwdo.F file is compiled, but at present the code in that module is not actually called.
With this commit, the UGWP physics can now be used in MPAS-A simulations
* Setting the 'config_gwdo_scheme' namelist option to 'bl_ugwp_gwdo' selects the
GSL GWDO scheme.
NOTE: If the GSL GWDO scheme is selected, the 'ugwp_oro_data_in' input stream
must specify a valid input file with UGWP static fields.
* Setting the 'config_ngw_scheme' namelist option to 'true' will activate the
non-stationary gravity wave drag scheme.
NOTE: If the NGW scheme is activated, the 'ugwp_ngw_in' input stream
must specify a valid input file.
* Setting the 'config_ugwp_diags' namelist option to 'true' will compute UGWP
diagnostic fields, which will be written by the 'diag_ugwp' output stream.
… GSL_UGWP_mdtoy
This port prepares for further changes and optimizations to follow.
…ke file. git commit b7f10af (PR 1242) added this file. The file was added to the Makefile, but it wasn't added to the cmake file. That resulted in a compile error when building with cmake: MPAS/src/core_atmosphere/physics/mpas_atmphys_driver.F:26:6: 26 | use mpas_atmphys_sfc_diagnostics,only: atmphys_sfc_diagnostics | 1 Fatal Error: Cannot open module file 'mpas_atmphys_sfc_diagnostics.mod' for reading at (1): No such file or directory compilation terminated.
Although the return error codes from calls to various functions in parse_reg_xml were saved to a local variable, the parse_reg_xml function always returned 0. As a result, any errors were effectively masked, and the registry 'parse' program always appeared to complete successfully. This commit modifies the Registry parse_reg_xml so that the return error codes from functions are checked, and if non-zero, cause the parse_reg_xml function to return a non-zero error code to the main program.
The registry-generated package-logic code for packages that employ an active_when attribute now contains calls to mpas_log_write to record information about the activity of packages to a core's log file. For example, the following messages are now written to the log file if two packages, foo and bar, were active and inactive, respectively: Configuring registry-specified packages... foo : T bar : F ----- done configuring registry-specified packages -----
…DEBUG=true
To aid in debugging packages that employ the active_when attribute in the
Registry, this commit adds log messages to the registry-generated package-logic
code that are only written to a core's log file when MPAS is compiled with
DEBUG=true.
For example, if the following two packages were defined:
<package name="foo" active_when="config_do_physics"/>
<package name="bar" active_when="config_nsteps > 0 .and. .not. config_dryrun"/>
compiling with DEBUG=true would result in log messages that would look like the
following (assuming the indicated values for namelist options config_do_physics,
config_nsteps, and config_dryrun):
Configuring registry-specified packages...
foo is active when (config_do_physics)
namelist settings:
------------------
config_do_physics = T
foo : T
bar is active when (config_nsteps > 0 .and. .not. config_dryrun)
namelist settings:
------------------
config_nsteps = 100
config_dryrun = T
bar : F
----- done configuring registry-specified packages -----
In order for the init_atmosphere core to make use of the ability for package logic to be generated by the registry through the active_when XML attribute, this commit includes the generated code in 'setup_packages.inc' in the init_atm_core_interface module, and it also adds a call to the generated init_atm_setup_packages_when routine from the init_atm_setup_packages routine. Note that the call to the generated init_atm_setup_packages_when routine occurs before any of the hand-written logic in the init_atm_setup_packages routine. As a consequence, settings of packages in the generated logic will be superseded by settings from hand-written logic for those packages.
In order for the atmosphere core to make use of the ability for package logic to be generated by the registry through the active_when XML attribute, this commit includes the generated code in 'setup_packages.inc' in the atm_core_interface module, and it also adds a call to the generated atm_setup_packages_when routine from the atm_setup_packages routine. Note that the call to the generated atm_setup_packages_when routine occurs before any of the hand-written logic in the atm_setup_packages routine. As a consequence, settings of packages in the generated logic will be superseded by settings from hand-written logic for those packages.
…S-Dev#1321) This merge enables the package logic code that determines when a package is active to be automatically generated by the registry through the use of a new XML attribute for <package> elements, active_when. Prior to the changes in this merge, defining a new package in the Registry with a <package> element required code to be written (typically in a core's setup_packages routine) to determine whether that package was active. For simple conditions -- e.g., a package being active if and only if a logical namelist option is true -- the need to write boilerplate-type Fortran code to retrieve package and namelist pointers, check that those pointers are valid, and to finally set the package variable, was unnecessarily cumbersome. This merge introduces a new XML attribute, active_when, that may be added to <package> elements in the Registry, along with new functionality in the registry processor to automatically generate package-logic code (the equivalent to what would otherwise need to be written by hand in a core's setup_packages routine) at compile time. If provided, the active_when attribute contains a Fortran expression that evaluates to a logical, and that may use any literal constants and namelist options (which are assumed to begin with the string 'config'). The expression from the active_when attribute is essentially used verbatim in Fortran code to set the package variable. For example, consider a package named 'foo' that is active if and only if the namelist option 'config_use_physics' is true. This package might have the following definition in the Registry: <package name="foo" active_when="config_do_physics"/> Logically, the generated code is equivalent to: fooActive = ( config_do_physics ) As another example, consider the following package definition in the Registry: <package name="bar" active_when="config_nsteps > 0 .and. .not. config_dryrun"/> The generated Fortran logic would be: barActive = ( config_nsteps > 0 .and. .not. config_dryrun ) When the package's logic involves string literal values, those values must use single-quote characters in their Registry definition, rather than double-quotes; for example: <package name="blaz" active_when="trim(config_physics_suite) == 'mesoscale'"/> Note that in these examples, the package's active_when logic is a valid Fortran logical expression that makes uses of namelist options and literal constants. The registry-generated package-logic code for packages that employ an active_when attribute contains calls to mpas_log_write to record information about the activity of packages to a core's log file. For example, the following messages are written to the log file if two packages, foo and bar, were active and inactive, respectively: Configuring registry-specified packages... foo : T bar : F ----- done configuring registry-specified packages ----- To aid in debugging packages that employ the active_when attribute in the Registry, the registry-generated package-logic code also contains log messages that are only written to a core's log file when MPAS is compiled with DEBUG=true. For example, if the following two packages were defined: <package name="foo" active_when="config_do_physics"/> <package name="bar" active_when="config_nsteps > 0 .and. .not. config_dryrun"/> compiling with DEBUG=true would result in log messages that would look like the following (assuming the indicated values for namelist options config_do_physics, config_nsteps, and config_dryrun): Configuring registry-specified packages... foo is active when (config_do_physics) namelist settings: ------------------ config_do_physics = T foo : T bar is active when (config_nsteps > 0 .and. .not. config_dryrun) namelist settings: ------------------ config_nsteps = 100 config_dryrun = T bar : F ----- done configuring registry-specified packages ----- In order to make use of this functionality introduced in this merge, a core must make two changes in its mpas_<CORE>_core_interface module: 1) The module should include through a pre-processor directive the registry-generated package-logic code, which resides in a file named 'setup_packages.inc' in the core's inc/ directory; e.g., 2) The setup_packages routine in the module should make a call to the generated <CORE>_setup_packages_when routine, which is defined in the 'setup_packages.inc' file; e.g., call atm_setup_packages_when(configs, packages) This merge updates the init_atmosphere and atmosphere cores so that they can make use of code generated by the active_when package attribute. The call to the generated (init_)atm_setup_packages_when routine occurs before any of the hand-written logic in the (init_)atm_setup_packages routine. As a consequence, settings of packages in the generated logic will be superseded by settings from hand-written logic for those packages.
This merge updates the atmosphere core CMakeLists.txt file to include the file mpas_atmphys_sfc_diagnostics.F, which was introduced by the the merge in commit b7f10af. That merge included changes to the Makefile but it did not update the CMakeLists.txt file, leading to errors when building MPAS-A with CMake. * atmosphere/cmake_pr1242: This change adds mpas_atmphys_sfc_diagnostics.F to the atmosphere cmake file.
…rget With the introduction of build tests for the PnetCDF library in merge commit 9a073c2, builds of MPAS with DEBUG=true were broken for the 'intel' build target. Two of the options, '-check all' and '-fpe0' that were included in the definition of LDFLAGS_DEBUG for the 'intel' build target only apply to Fortran, while the test program for the PnetCDF library is pure C, resulting in the build-time error message: ********************************************************* ERROR: Test PnetCDF C program could not be compiled by mpicc. Please ensure you have a working PnetCDF library installed. The following compilation command failed with errors: mpicc pnetcdf.c -I<path to pnetcdf>/include -g -traceback -DSINGLE_PRECISION -g -check all -fpe0 -traceback -L<path to pnetcdf>/lib -lpnetcdf -o pnetcdf.out Test program pnetcdf.c and output pnetcdf.log have been left in the top-level MPAS directory for further debugging ********************************************************* The underlying error messages from the icx compiler identify the '-check' and '-fpe0' options as the source of the failure: icx: error: unknown argument: '-check' icx: error: unknown argument: '-fpe0' This commit removes '-check all' and '-fpe0' from the definition of LDFLAGS_DEBUG for the 'intel' target, and since '-check all' must be specified during both compilation and linking, the '-check all' option in FFLAGS_DEBUG is replaced with '-check bounds,pointers,arg_temp_created,format,shape,contiguous', which includes most of the checks provided by 'all', though omitting, most notably, checks for the use of uninitialized memory.
…scalar The module variable config_apply_lbcs in the atm_time_integration module was previously declared as a logical pointer; however, this module-level config_apply_lbcs variable is used in OpenACC parallel regions in several routines. Since OpenACC implicitly copies scalar variables but not pointers, this commit converts the module-level config_apply_lbcs variable to a simple logical scalar. Code in the atm_timestep routine for setting config_apply_lbcs now retrieves a local pointer variable, which is then assigned to config_apply_lbcs.
The 3-d refl10cm field is captured from 1-d columns of computed 10 cm radar reflectivity when either the Thompson, Thompson aerosol-aware, or WSM6 microphysics schemes is used. Prior to this commit, the 1-d columns of reflectivity were saved to a transposed refl10cm_p array in the compute_radar_reflectivity routine, and refl10cm_p was then copied to refl10cm in the microphysics_to_MPAS routine. Since the compute_radar_reflectivity subroutine already accesses other registry-defined fields through calls to mpas_pool_get_array, the refl10cm field can be accessed directly in that subroutine, eliminating the need for the intermediate refl10cm_p array.
…ev#1327) This merge fixes DEBUG build issues due to the use of Fortran-only LDFLAGS in the 'intel' build target. With the introduction of build tests for the PnetCDF library in merge commit 9a073c2, builds of MPAS with DEBUG=true were broken for the 'intel' build target. Two of the options, '-check all' and '-fpe0' that were included in the definition of LDFLAGS_DEBUG for the 'intel' build target only apply to Fortran, while the test program for the PnetCDF library is pure C, resulting in the build-time error message: ********************************************************* ERROR: Test PnetCDF C program could not be compiled by mpicc. Please ensure you have a working PnetCDF library installed. The following compilation command failed with errors: mpicc pnetcdf.c -I<path to pnetcdf>/include -g -traceback -DSINGLE_PRECISION -g -check all -fpe0 -traceback -L<path to pnetcdf>/lib -lpnetcdf -o pnetcdf.out Test program pnetcdf.c and output pnetcdf.log have been left in the top-level MPAS directory for further debugging ********************************************************* The underlying error messages from the icx compiler identify the '-check' and '-fpe0' options as the source of the failure: icx: error: unknown argument: '-check' icx: error: unknown argument: '-fpe0' This merge removes '-check all' and '-fpe0' from the definition of LDFLAGS_DEBUG for the 'intel' target, and since '-check all' must be specified during both compilation and linking, the '-check all' option in FFLAGS_DEBUG is replaced with '-check bounds,pointers,arg_temp_created,format,shape,contiguous', which includes most of the checks provided by 'all', though omitting, most notably, checks for the use of uninitialized memory. * framework/makefile_intel_debug: Fix DEBUG build issues due to Fortran-only LDFLAGS for the 'intel' target
MPAS-Dev#1328) This merge converts the config_apply_lbcs variable from a 'logical, pointer' to a simple 'logical' scalar variable. The module variable config_apply_lbcs in the atm_time_integration module was previously declared as a 'logical, pointer' variable; however, this module-level config_apply_lbcs variable is used in OpenACC parallel regions in several routines. Since OpenACC implicitly copies scalar variables but not pointers, this merge converts the module-level config_apply_lbcs variable to a simple 'logical' scalar. Code in the atm_timestep routine for setting config_apply_lbcs now retrieves a local pointer variable, which is then assigned to config_apply_lbcs. * atmosphere/nonpointer_config_apply_lbcs: Convert config_apply_lbcs from a logical pointer to a simple logical scalar
…#1329) This merge eliminates the intermediate refl10cm_p array when capturing the 3-d refl10cm field. The 3-d refl10cm field is captured from 1-d columns of computed 10 cm radar reflectivity when either the Thompson, Thompson aerosol-aware, or WSM6 microphysics schemes is used. Prior to this merge, the 1-d columns of reflectivity were saved to a transposed refl10cm_p array in the compute_radar_reflectivity routine, and refl10cm_p was then copied to refl10cm in the microphysics_to_MPAS routine. Since the compute_radar_reflectivity subroutine already accesses other registry-defined fields through calls to mpas_pool_get_array, the refl10cm field can be accessed directly in that subroutine, eliminating the need for the intermediate refl10cm_p array. * atmosphere/remove_refl10cm_p: Eliminate intermediate refl10cm_p array when capturing 3-d refl10cm
The setup_run_dir.py script, located in the testing_and_setup/atmosphere
directory, may be used to set up a run directory for MPAS-A by copying and
linking executables, namelist files, streams files, and physics look-up tables
from the MPAS-Model directory to a specified directory.
The usage of the setup_run_dir.py script is as follows:
usage: setup_run_dir.py [-h] [-o] [-i | -a] run_dir
positional arguments:
run_dir the run directory to set up
options:
-h, --help show this help message and exit
-o, --overwrite overwrite any existing files and symbolic links
-i, --init-only only set up files for the init_atmosphere core
-a, --atm-only only set up files for the atmosphere core
Note: The setup_run_dir.py script assumes that the top-level MPAS-Model
directory is located two directory levels above the script (i.e., at ../..
relative to the script location).
This merge adds a new Python script for setting up an MPAS-Atmosphere run
directory.
The setup_run_dir.py script, located in the testing_and_setup/atmosphere
directory, may be used to set up a run directory for MPAS-A by copying and
linking executables, namelist files, streams files, and physics look-up tables
from the MPAS-Model directory to a specified directory.
The usage of the setup_run_dir.py script is as follows:
usage: setup_run_dir.py [-h] [-o] [-i | -a] run_dir
positional arguments:
run_dir the run directory to set up
options:
-h, --help show this help message and exit
-o, --overwrite overwrite any existing files and symbolic links
-i, --init-only only set up files for the init_atmosphere core
-a, --atm-only only set up files for the atmosphere core
* atmosphere/setup_run_dir:
Add new Python script for setting up an MPAS-Atmosphere run directory
…lars_mono_work When compiled with OpenACC enabled using the nvfortran compiler, running with 'config_split_dynamics_transport = false' in the namelist.atmsophere file results in a runtime error for rho_zz_int: FATAL ERROR: variable in data clause is partially present on the device: name=rho_zz_int(:,:) file:src/core_atmosphere/dynamics/mpas_atm_time_integration.F atm_advance_scalars_mono_work line:3942 The rho_zz_int array is only used if local_advance_density is .true. in the atm_advance_scalars_mono_work routine, and local_advance_density is true if and only if config_split_dynamics_transport is true. Consequently, simulations with 'config_split_dynamics_transport = false' in the namelist only allocate the rho_zz_int array with dimensions (1,1), rather than the full dimensions (nVertLevels,nCells+1) that are required when local_advance_density is true. This difference in the allocated dimensions of rho_zz_int apparently causes problems for the OpenACC runtime, which expects rho_zz_int to be allocated as indicated by the explicit-shape specification, dimension(nVertLevels,nCells+1), for the rho_zz_int dummy argument to atm_advance_scalars_mono_work. To work around the partially present error, this commit changes the rho_zz_int dummy argument to a deferred shape array, with dimension(:,:).
) This merge fixes an OpenACC "partially present" error for rho_zz_int in the atm_advance_scalars_mono_work routine. When compiled with OpenACC enabled using the nvfortran compiler, running with 'config_split_dynamics_transport = false' in the namelist.atmsophere file results in a runtime error for rho_zz_int: FATAL ERROR: variable in data clause is partially present on the device: name=rho_zz_int(:,:) file:src/core_atmosphere/dynamics/mpas_atm_time_integration.F atm_advance_scalars_mono_work line:3942 The rho_zz_int array is only used if local_advance_density is true in the atm_advance_scalars_mono_work routine, and local_advance_density is true if and only if config_split_dynamics_transport is true. Consequently, simulations with 'config_split_dynamics_transport = false' in the namelist only allocate the rho_zz_int array with dimensions (1,1), rather than the full dimensions (nVertLevels,nCells+1) that are required when local_advance_density is true. This difference in the allocated dimensions of rho_zz_int apparently causes problems for the OpenACC runtime, which expects rho_zz_int to be allocated as indicated by the explicit-shape specification, dimension(nVertLevels,nCells+1), for the rho_zz_int dummy argument to atm_advance_scalars_mono_work. To work around the "partially present" error, this merge changes the rho_zz_int dummy argument to a deferred shape array, with dimension(:,:). * atmosphere/rho_zz_int_decl: Fix OpenACC partially present error for rho_zz_int in atm_advance_scalars_mono_work
In order to run test scripts outside of a testing framework, handling of the environment setup should not be solely dependent on running within a dedicated test framework. This has the added benefit of compartmentalizing the duties of environment and dependency solving from running the tests. These environment scripts allow for the selection of a particular environment with the default being the fqdn of the current host. From there, arguments are routed using standard POSIX-sh to a respective script. In the case of Derecho (applicable to any system using lmod) all subsequent argument are treated as modules to load into the current session. The hostenv.sh script relies on one "argument" $AS_HOST being passed in via variable setting to facilitate selection. The helpers.sh script provides convenience features for substring checking in sh, delayed environment variable expansion via eval, and quick banner creation. The derecho.sh script is included as the first supported environment.
This script will facilitate the first tests. There are only three requirements of any given test script with the planned testing framework If a different testing framework is used in the future, these requirements of the test scripts can and should be re-evaluated. The test script should : 1. Take the intended host / configuration env as the first argument 2. Take the working directory which we cd to as the second argument 3. Output some key phrase at the end of the test to denote success, anything else (non-zero exit code, no phrase but return zero) is considered failure This particular compilation test script satisfies the above while also providing enough flexibility to select core, target, parallel jobs, and other command-line options into the make build. Additionally, for convenience environment variables can be passed in as command-line options to the test script to modularize certain inputs.
Following the documentation of the hpc-workflows testing framework and the testing structure found in .ci/, a JSON file for a GNU compilation test was added. This test will compile the atmosphere core using gnu + single precision. If this test is run using the derecho configuration the appropriate modules will attempt to be loaded. For non-derecho environments, per the testing structure under .ci/, if no configuration exists in .ci/hostenv.sh then the current environment will be used verbatim.
some changes to tests config some scripts to support performance benchmarking adding nvhpc-cpu some changes to exit codes
Adding reference generation capability and a separate jsonc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on MPAS-Dev#1284 to add runtime tests for MPAS.
To launch tests from the root directory
The following toolchain/builds are proposed
gnu,gnu-double,gpu-openmp,gnu-debugnvhpc-cpu,nvhpc-gpu,nvhpc-gpu-doubleintel,intel-doubleThe following test scenarios are proposed:
Jablonowski-Williamson baroclinic wave (dycore)Limited area over CONUSAquaplanetSquall lineThe following tests are proposed, orthogonal to test scenarios above.
Base: The simplest test configuration for a given scenario. The output restart file is checked for bit identical results against a stored reference.Restart: Tests the ability to proceed from an intermediate restart file (provided by base run) and produce results bit-identical to base run.Parallel: Tests for various parallel implementations/configurations. The resulting restart files are checked for bit-identical results against the base runs. This is further classified intoMPIOMPMultinode CPUMulti GPUMultinode GPUPerformance: Five instances of a shorter run are performed and the timing statistics are compared against references stored in a database. The idea would be to run a nightly build + performance runs for the develop branch, and store the statistics in the SQLite databse. When the performance check is run for PRs, then we could use the average of the last 5-10 days to decide if the performance of our PR has regressed.Test matrix:
The proposal is to avoid running all possible combinations of (toolchain X scenario X tests) and instead carefully map out a minimum set of tests in this space that would provide a reasonable guarantee of correctness and keep a eye on performance regression.
GNU (OPTIMIZED)GNU (DEBUG)- Build onlyGNU - DOUBLE PRECISION (OPTIMIZED)NVHPC-CPU (OPTIMIZED)NVHPC-GPU (OPTIMIZED)NVHPC-GPU DOUBLE (OPTIMIZED)INTEL (OPTIMIZED)INTEL (DEBUG)- Build onlyReference generation
To periodically update the reference restart files against which the base runs are compare, we could use another json file
.ci/mpas_refgen.jsoncto generate these references from the base branch (develop). This would be triggered manually as required, when either the base branch introduces substantial changes or the GPU porting of a routine results in non BFB identical results (which have been otherwise verified).To launch the reference generation from the root directory