diff --git a/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/README.md b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/README.md new file mode 100644 index 000000000..a98f63ac4 --- /dev/null +++ b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/README.md @@ -0,0 +1,157 @@ +# Building and Running OPM Flow on OCI GPU infrastructure + +## The OPM initiative + +The [Open Porous Media (OPM)](https://opm-project.org/) initiative is an innovative open source project for modeling and simulating porous media processes. + + ![OPM Flow official illustration](/assets/images/OPM-Flow.png) + +OPM Flow is a fully-implicit, black-oil simulator capable of running industry-standard simulation models. The simulator is implemented using automatic differentiation to enable rapid development of new fluid models. + +## Configuring a GPU instance on OCI + +Oracle Cloud Infrastructure offers GPU shapes for all kinds of workloads, including: +- NVIDIA A10 24GB (available on-demand) + - VM with 1 or 2 GPUs + - BM with 4 GPUs +- NVIDIA A100 40GB (available on-demand) + - VM with 1 GPU + - BM with 8 GPUs, NVLink and RDMA cluster networking +- NVIDIA L40S 48GB + - BM with 4 GPUs and RDMA cluster networking +- NVIDIA A100 80GB + - VM with 1 GPU + - BM with 8 GPUs, NVLink and RDMA cluster networking +- NVIDIA H100 80GB + - BM with 8 GPUs, NVLink and RDMA cluster networking +- AMD MI300X 192GB + - BM with 8 GPUs, HIP and RDMA cluster networking +This tutorial covers NVIDIA GPU shapes only but can be easily adapted for AMD GPU shapes. A single A100 80 GB virtual machine (VM.GPU.A100.80GB.1) with a Cannonical Ubuntu 22.04 image is used. + +It is necessary to install the NVIDIA CUDA Compiler (NVCC) to build Flow: +``` +sudo apt-get install -y nvidia-cuda-toolkit +``` + +## Installing OPM Flow on OCI GPU infrastructure + +The project can be installed from packages or built from sources. For GPU support, building from sources is mandatory. + +### Installing from binaries + +When GPU are not required or not available, OPM Flow can run on CPU only. In this case, installing the toolset from packages is the easiest way (see tutorial [here](https://opm-project.org/?page_id=245)). + +### Building from sources + +Running OPM Flow on GPU requires to build the toolset from sources in order to setup the GPU support. Common Building from Source guidelines are available [here](https://opm-project.org/?page_id=231). In the sequel, one can get more hands-on commands and advices for a complete configuration. + +#### Installing prerequisites + +A collection of libraries, framework and tools are necessary to build Flow. A complete description is available [here](https://opm-project.org/?page_id=239).They can be installed using the `install_prerequisites.sh` script. +``` +sudo ./assets/scripts/install_prerequisites.sh +``` + +#### Building the modules + +4 modules must be built in a specific order: +1. `opm-common` +2. `opm-grid` +3. `opm-simulators` +4. `opm-upscaling` + +For `opm-common`and `opm-grid`, no specific actions is required. Simply follow the build instructions: +``` +git clone https://github.com/OPM/[modulename].git +mkdir [modulename]/build && cd [modulename]/build +cmake .. +make +``` +However, for `opm-simulators` it is necessary to turn off the `USE_GPU_BRIDGE` option at the `cmake` stage: +``` +git clone https://github.com/OPM/opm-simulators.git +mkdir opm-simulators/build && cd opm-simulators/build +cmake .. -DUSE_GPU_BRIDGE=OFF +make +``` +The following error might occur at the `make` stage: +``` +/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’: + 435 | function(_Functor&& __f) + | ^ +/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’ +/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’: + 530 | operator=(_Functor&& __f) + | ^ +/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’ +make[2]: *** [CMakeFiles/opmsimulators.dir/build.make:2680: CMakeFiles/opmsimulators.dir/opm/simulators/linalg/gpuistl/detail/vector_operations.cu.o] Error 1 +make[2]: *** Waiting for unfinished jobs.... +make[1]: *** [CMakeFiles/Makefile2:530: CMakeFiles/opmsimulators.dir/all] Error 2 +make: *** [Makefile:146: all] Error 2 +``` +If it does, two lines (beginning by the `noexcept` keyword) must be commented in the `std_function.h` file, i.e.: +- approximately line 430: +``` + template>> + function(_Functor&& __f) + //noexcept(_Handler<_Functor>::template _S_nothrow_init<_Functor>()) // CUDA does not support this line + : _Function_base() + +``` +- approximately line 450: +``` + template>> + function(_Functor&& __f) + //noexcept(_Handler<_Functor>::template _S_nothrow_init<_Functor>()) // CUDA does not support this line + : _Function_base() + +``` +Finally, build `opm-upscaling`: +``` +git clone https://github.com/OPM/opm-upscaling.git +mkdir opm-upscaling/build && cd opm-upscaling/build +cmake .. +make +``` + +## Running OPM Flow on GPU + +Running the Flow simulator on one or more GPUs requires to use GPU-specific solver and preconditionners. They can be defined in a linear solver [file](/assets/scripts/gpu-solver.json) in the JSON format, for example: +``` +{ + "tol": "0.01", + "maxiter": "200", + "verbosity": "0", + "solver": "gpubicgstab", + "preconditioner": { + "type": "GPUDILU", + "verbosity" : 0, + "split_matrix": "true", + "tune_gpu_kernels": "true", + "mixed_precision_scheme": 1 + } +} +``` +To run the simulation, open datasets are available [here](git clone https://github.com/OPM/opm-data.git). For example, to run the Norne use case, simply execute: +``` +/home/ubuntu/opm-simulators/build/bin/flow NORNE_ATW2013.DATA --output-dir=out_gpu --matrix-add-well-contributions=true --linear-solver=/home/ubuntu/gpu-solver.json +``` +for 1 GPU and: +``` +mpirun -np N /home/ubuntu/opm-simulators/build/bin/flow NORNE_ATW2013.DATA --output-dir=out_gpu --matrix-add-well-contributions=true --threads-per-process=1 --linear-solver=/home/ubuntu/gpu-solver.json +``` +for N GPUs. + +## Notes: + +Here are a few comments to take into account when considering running Flow on GPUs: +* The options `--matrix-add-well-contributions=true` and `--threads-per-process=1` are recommended by OPM. +* Because of the import CPU/GPU traffic, running Flow on GPU is only relevant for models above a certain size (few hundreds of thousands of cells). +* Using a CUDA-aware version of Open MPI may improve the overall performance of the simulation. + +## External Links + +* [The Open Porous Media Initiative](https://opm-project.org/) +* [Building CUDA-aware Open MPI](https://www.open-mpi.org/faq/?category=buildcuda) \ No newline at end of file diff --git a/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/OPM-Flow.png b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/OPM-Flow.png new file mode 100644 index 000000000..5c2f9e6e3 Binary files /dev/null and b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/OPM-Flow.png differ diff --git a/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/SLEIPNER_ORG_3D_View_SGAS_170_01_Dec_2010_FILTER.png b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/SLEIPNER_ORG_3D_View_SGAS_170_01_Dec_2010_FILTER.png new file mode 100644 index 000000000..e2f993038 Binary files /dev/null and b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/SLEIPNER_ORG_3D_View_SGAS_170_01_Dec_2010_FILTER.png differ diff --git a/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/build_flow_gpu.sh b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/build_flow_gpu.sh new file mode 100644 index 000000000..69155bd66 --- /dev/null +++ b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/build_flow_gpu.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +for repo in opm-common opm-grid +do + git clone https://github.com/OPM/$repo.git + mkdir $repo/build && cd $repo/build + cmake .. + make -j 8 + cd ../.. +done + +git clone https://github.com/OPM/opm-simulators.git +mkdir opm-simulators/build && cd opm-simulators/build +cmake .. -DUSE_GPU_BRIDGE=OFF +make -j 8 +cd ../.. + +git clone https://github.com/OPM/opm-upscaling.git +mkdir opm-upscaling/build && cd opm-upscaling/build +cmake .. +make -j 8 +cd ../.. + + + +/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’: + 435 | function(_Functor&& __f) + | ^ +/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’ +/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’: + 530 | operator=(_Functor&& __f) + | ^ +/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’ +make[2]: *** [CMakeFiles/opmsimulators.dir/build.make:2680: CMakeFiles/opmsimulators.dir/opm/simulators/linalg/gpuistl/detail/vector_operations.cu.o] Error 1 +make[2]: *** Waiting for unfinished jobs.... +make[1]: *** [CMakeFiles/Makefile2:530: CMakeFiles/opmsimulators.dir/all] Error 2 +make: *** [Makefile:146: all] Error 2 \ No newline at end of file diff --git a/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/gpu-solver.json b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/gpu-solver.json new file mode 100644 index 000000000..a5cd48080 --- /dev/null +++ b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/gpu-solver.json @@ -0,0 +1,13 @@ +{ + "tol": "0.01", + "maxiter": "200", + "verbosity": "0", + "solver": "gpubicgstab", + "preconditioner": { + "type": "GPUDILU", + "verbosity" : 0, + "split_matrix": "true", + "tune_gpu_kernels": "true", + "mixed_precision_scheme": 1 + } +} \ No newline at end of file diff --git a/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/install_prerequisites.sh b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/install_prerequisites.sh new file mode 100644 index 000000000..699a6224d --- /dev/null +++ b/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/install_prerequisites.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +# Make sure we have updated URLs to packages etc. +sudo apt-get update -y + +# For server edition of Ubuntu add-apt-repository depends on +sudo apt-get install -y software-properties-common + +# Add PPA for OPM packages +sudo add-apt-repository -y ppa:opm/ppa +sudo apt-get update -y + +# Packages necessary for building +sudo apt-get install -y build-essential gfortran pkg-config cmake + +# Packages necessary for documentation +sudo apt-get install -y doxygen ghostscript texlive-latex-recommended gnuplot + +# Packages necessary for version control +sudo apt-get install -y git-core + +# MPI for parallel programs +sudo apt-get install -y mpi-default-dev + +# Prerequisite libraries +sudo apt-get install -y libblas-dev libboost-all-dev libsuitesparse-dev libtrilinos-zoltan-dev libfmt-dev libcjson-dev + +# Parts of Dune needed +sudo apt-get install -y libdune-common-dev libdune-geometry-dev libdune-istl-dev libdune-grid-dev \ No newline at end of file