Compiling

The codebase uses a manually written Makefile located in the top-level directory that has been tested on the Della HPC system. To build the project, you can use standard make commands. Additionally, make sure that HDF5 and NETCDF modules or paths are defined before building.

Before compiling, it is strongly recommended that you clean any previously built files to ensure a fresh and consistent build:

make clean

Load the required modules:

module load cudatoolkit/12.9
module load openmpi/gcc/4.1.6
module load hdf5/gcc/openmpi-4.1.6/1.14.4
module load netcdf/gcc/hdf5-1.14.4/openmpi-4.1.6/4.9.2

Then compile the project from the root:

make

This will generate the final executable:

bin/runoff

Compiler and Build Flags

This project uses NVIDIA’s CUDA Compiler (nvcc) along with the GNU C++17 standard for host code. The build process also integrates MPI for parallel execution and links against NetCDF and Boost libraries.

Default (Release) Build

By default, the Makefile compiles with high optimization and GPU support:

NVCCFLAGS := -std=c++17 -rdc=true \
  -gencode arch=compute_90,code=sm_90 \
  -O2 $(DEVICE_INCLUDES) -DUSE_MODEL_204 -Wno-deprecated-gpu-targets
HOSTFLAGS := -std=c++17 $(HOST_INCLUDES) -DUSE_MODEL_204 \
             -L${NETCDF_PATH}/lib64 -lnetcdf

These flags enable:

• GPU support with architecture-specific optimizations (sm_90)
• Link-time device code generation (-rdc=true)
• Model-specific optimizations via -DUSE_MODEL_204
• NetCDF linking for I/O functionality

Debug Build (DEBUG=1)

When DEBUG=1 is set, the compiler switches to debugging mode:

NVCCFLAGS += -g -G -O0

This disables optimizations, generates debugging symbols, and allows device-level debugging of CUDA kernels.

Verbose Build (VERBOSE=1)

When VERBOSE=1 is specified, the Makefile prints full compilation and linking commands to the console. Without this flag, command output is suppressed for cleaner logs.

MPI and NetCDF

• MPI is automatically detected and its include/link flags are appended (MPI_INC, MPI_LIB).
• The build links against the NetCDF library, and you must ensure NETCDF_PATH is set correctly to a valid NetCDF installation.

Usage

You can combine these options when building:

make                   # Default optimized build
make DEBUG=1           # Build with CUDA debugging enabled
make VERBOSE=1         # Show all compiler/linker commands
make DEBUG=1 VERBOSE=1 # Debug build with verbose output

Source Organization

The code is organized into several directories, and the Makefile automatically compiles all source files under src/, including nested directories like src/I_O and src/utils.

Object files are stored in a separate build/ directory, and the final binary is placed in the bin/ directory:

src/          → Source files  
build/        → Object files  
bin/          → Final binary  

Compatibility

This project is designed for heterogeneous HPC environments and requires both MPI and CUDA support. It should build successfully on systems that provide:

• C++17 support for host code
• CUDA Toolkit (with nvcc) for GPU acceleration
• MPI for distributed computation
• NetCDF library for input/output operations

Supported Compilers

The default build uses nvcc (with GNU g++ as the host compiler).
CUDA compilation requires nvcc or a compiler with full CUDA support.
For MPI + CUDA builds, the following setups are recommended:

• NVIDIA HPC SDK (nvc++) – Full support for CUDA and MPI.
• GNU g++ with nvcc – Standard and widely tested.

Recommendations

• Use the latest CUDA Toolkit and MPI implementation available on your system.
• Ensure that the target GPU architecture (sm_90 by default) matches your hardware; adjust the -gencode flags in the Makefile if needed.
• The code requires CUDA-enabled hardware; no CPU-only fallback is provided.