Cray Fortran Compatibility

[sleigh2022]

Summary

HPE Cray supercomputers provide the native crayftn Fortran compiler for building executables that are optimised to Cray hardware. The compiler has notably stricter coding standards than other compilers such as gfortran. Previous versions of ReMKiT1D (v1.1.0) were reported to be crayftn compatible, but compatibility has been lost since.

While it is possible to use gfortran on Cray systems, it would be ideal to optimise ReMKiT1D for production runs in the future.

To-do (being updated)

See below comments for descriptions of these issues:

• 1. Restructure interface-procedure pattern in assertion_utility.f90 (and others if applicable).
• 2. Restructure CVODEIntegrator, ensuring function checked by FCVODEInit and FCVBBDPrecInit exists at module scope and not subroutine scope.

Repro steps (ARCHER2 Supercomputer)

Using the default environment on ARCHER2, where Cray PE (compilers), Cray-mpich (MPI) and Cray-libsci (BLAS and LAPACK) are pre-loaded, set a work directory, and load the preinstalled CMake and Cray-HDF5:

# ReMKiT1D and its dependencies live here
export RMKROOT=<enter your path here>

# Use Cray compilers and the pre-installed version of CMake
module load cmake/3.18.4 cray-hdf5/1.12.2.7
export FC=ftn
export CC=cc

# Subdirectory for dependencies
REQ_LIBS_PATH=$RMKROOT/remkit1d_reqs_cray
mkdir $REQ_LIBS_PATH
cd $REQ_LIBS_PATH

# A further subdirectory for dependency installs
mkdir lib_installs

Build and configure PETSc, build and install SUNDIALS and json-fortran, then ReMKiT1D.

#!/bin/bash

# Install and configure PETSc with hypre
git clone -b release-3.17 https://gitlab.com/petsc/petsc.git petsc
cd petsc
./configure --download-hypre=1 PETSC_ARCH=arch-opt-hypre --with-debugging=no COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
make PETSC_DIR=$REQ_LIBS_PATH/petsc PETSC_ARCH=arch-opt-hypre all

cd $REQ_LIBS_PATH

# Install SUNDIALS
git clone https://github.com/LLNL/sundials.git
cd sundials
mkdir build
cd build
cmake \
    -DENABLE_MPI=ON \
    -DCMAKE_INSTALL_PREFIX=$REQ_LIBS_PATH/lib_installs/sundials \
    -DENABLE_LAPACK=ON \
    -DLAPACK_LIBRARIES="${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_cray_mpi.so" \
    -DBUILD_FORTRAN_MODULE_INTERFACE=ON \
    -DCMAKE_Fortran_COMPILER=$FC \
    -DCMAKE_C_COMPILER=$CC \
    ..
make install

cd $REQ_LIBS_PATH

# Install json-fortran
git clone -b 8.2.5 https://github.com/jacobwilliams/json-fortran.git
cd json-fortran
mkdir build
cd build
cmake .. -DCMAKE_Fortran_COMPILER=$FC -DCMAKE_INSTALL_PREFIX=$REQ_LIBS_PATH/lib_installs/json-fortran -DSKIP_DOC_GEN=TRUE
make install

cd $RMKROOT

# Build ReMKiT1D itself - The flag -eZ enables pre-processor statements
git clone -b v1.3.0-branch https://github.com/ukaea/ReMKiT1D.git
cd ReMKiT1D
mkdir build
cd build
cmake .. -DCMAKE_Fortran_FLAGS="-eZ"
make
export RMKEXE=$RMKROOT/ReMKiT1D/build/src/executables/ReMKiT1D/ReMKiT1D

[sleigh2022]

1. Interface and duplicate subroutine definitions

The following interface-procedure pattern which is typically used to provide polymorphism is gfortran-compatible but not Cray-compatible:

module mymod
  interface mysub
    subroutine mysub(i)
      integer :: i
    end subroutine
  end interface
contains
  subroutine mysub(i)
    integer :: i
    print *, "Hello", i
  end subroutine
end module

Solution: module procedure (subroutine names) where every uniquely-named subroutine is declared in module scope, and is then associated with an interface (also with a unique name) through a module procedure call and not another subroutine with the same name. For example for a polymorphic interface mysub:

module mymod
contains
  subroutine mysub_int(i)
    integer :: i
    print *, "Integer version", i
  end subroutine

  subroutine mysub_real(r)
    real :: r
    print *, "Real version", r
  end subroutine

  interface mysub
    module procedure mysub_int, mysub_real
  end interface
end module

One module known to have the problematic pattern is assertion_utility.f90 but others may also have this issue too.

[sleigh2022]

2. Functions with bind(C) cannot exist inside of other subroutines

crayftn only allows functions and subroutines with C bindings to exist at the module scope level. This is a strict requirement by the compiler and cannot be disabled with flags. The object-oriented CVODEIntegrator class is largely based on the official example from SUNDIALS, which is non-OOP.

Here:

• Functions firhs and LocalgFn are the RHS implicit functions,
• Each are passed into c_funloc to get their C address, which is then passed into FCVodeInit and FCVBBDPrecInit to initialise the solver and preconditioner.
• Both functions make use of bind(C) and C variables,
• In ReMKiT1D, their equivalents are rhs and rhsGfun respectively, but both are contained inside subroutine CVODEIntegrator instead of at module scope as required.
• At the same time rhs requires data from the CVODEIntegrator class which is not C-operable (hence why it was initially in CVODEIntegrator's scope).

To resolve this issue, the functionality of rhs that use data from the CVODEIntegrator class might need to be moved from rhs and into CVODEIntegrator.