Skip to content

Commit 026dc57

Browse files
yanncaniouoracleyanncaniouoracle
committed
First commit
1 parent a3d1810 commit 026dc57

File tree

6 files changed

+236
-0
lines changed

6 files changed

+236
-0
lines changed

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/README.md

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Building and Running OPM Flow on OCI GPU infrastructure
2+
3+
## The OPM initiative
4+
5+
The [Open Porous Media (OPM)](https://opm-project.org/) initiative is an innovative open source project for modeling and simulating porous media processes.
6+
7+
![OPM Flow official illustration](/assets/images/OPM-Flow.png)
8+
9+
OPM Flow is a fully-implicit, black-oil simulator capable of running industry-standard simulation models. The simulator is implemented using automatic differentiation to enable rapid development of new fluid models.
10+
11+
## Configuring a GPU instance on OCI
12+
13+
Oracle Cloud Infrastructure offers GPU shapes for all kinds of workloads, including:
14+
- NVIDIA A10 24GB (available on-demand)
15+
- VM with 1 or 2 GPUs
16+
- BM with 4 GPUs
17+
- NVIDIA A100 40GB (available on-demand)
18+
- VM with 1 GPU
19+
- BM with 8 GPUs, NVLink and RDMA cluster networking
20+
- NVIDIA L40S 48GB
21+
- BM with 4 GPUs and RDMA cluster networking
22+
- NVIDIA A100 80GB
23+
- VM with 1 GPU
24+
- BM with 8 GPUs, NVLink and RDMA cluster networking
25+
- NVIDIA H100 80GB
26+
- BM with 8 GPUs, NVLink and RDMA cluster networking
27+
- AMD MI300X 192GB
28+
- BM with 8 GPUs, HIP and RDMA cluster networking
29+
This tutorial covers NVIDIA GPU shapes only but can be easily adapted for AMD GPU shapes. A single A100 80 GB virtual machine (VM.GPU.A100.80GB.1) with a Cannonical Ubuntu 22.04 image is used.
30+
31+
It is necessary to install the NVIDIA CUDA Compiler (NVCC) to build Flow:
32+
```
33+
sudo apt-get install -y nvidia-cuda-toolkit
34+
```
35+
36+
## Installing OPM Flow on OCI GPU infrastructure
37+
38+
The project can be installed from packages or built from sources. For GPU support, building from sources is mandatory.
39+
40+
### Installing from binaries
41+
42+
When GPU are not required or not available, OPM Flow can run on CPU only. In this case, installing the toolset from packages is the easiest way (see tutorial [here](https://opm-project.org/?page_id=245)).
43+
44+
### Building from sources
45+
46+
Running OPM Flow on GPU requires to build the toolset from sources in order to setup the GPU support. Common Building from Source guidelines are available [here](https://opm-project.org/?page_id=231). In the sequel, one can get more hands-on commands and advices for a complete configuration.
47+
48+
#### Installing prerequisites
49+
50+
A collection of libraries, framework and tools are necessary to build Flow. A complete description is available [here](https://opm-project.org/?page_id=239).They can be installed using the `install_prerequisites.sh` script.
51+
```
52+
sudo ./assets/scripts/install_prerequisites.sh
53+
```
54+
55+
#### Building the modules
56+
57+
4 modules must be built in a specific order:
58+
1. `opm-common`
59+
2. `opm-grid`
60+
3. `opm-simulators`
61+
4. `opm-upscaling`
62+
63+
For `opm-common`and `opm-grid`, no specific actions is required. Simply follow the build instructions:
64+
```
65+
git clone https://github.com/OPM/[modulename].git
66+
mkdir [modulename]/build && cd [modulename]/build
67+
cmake ..
68+
make
69+
```
70+
However, for `opm-simulators` it is necessary to turn off the `USE_GPU_BRIDGE` option at the `cmake` stage:
71+
```
72+
git clone https://github.com/OPM/opm-simulators.git
73+
mkdir opm-simulators/build && cd opm-simulators/build
74+
cmake .. -DUSE_GPU_BRIDGE=OFF
75+
make
76+
```
77+
The following error might occur at the `make` stage:
78+
```
79+
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
80+
435 | function(_Functor&& __f)
81+
| ^
82+
/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’
83+
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
84+
530 | operator=(_Functor&& __f)
85+
| ^
86+
/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’
87+
make[2]: *** [CMakeFiles/opmsimulators.dir/build.make:2680: CMakeFiles/opmsimulators.dir/opm/simulators/linalg/gpuistl/detail/vector_operations.cu.o] Error 1
88+
make[2]: *** Waiting for unfinished jobs....
89+
make[1]: *** [CMakeFiles/Makefile2:530: CMakeFiles/opmsimulators.dir/all] Error 2
90+
make: *** [Makefile:146: all] Error 2
91+
```
92+
If it does, two lines (beginning by the `noexcept` keyword) must be commented in the `std_function.h` file, i.e.:
93+
- approximately line 430:
94+
```
95+
template<typename _Functor,
96+
typename _Constraints = _Requires<_Callable<_Functor>>>
97+
function(_Functor&& __f)
98+
//noexcept(_Handler<_Functor>::template _S_nothrow_init<_Functor>()) // CUDA does not support this line
99+
: _Function_base()
100+
101+
```
102+
- approximately line 450:
103+
```
104+
template<typename _Functor,
105+
typename _Constraints = _Requires<_Callable<_Functor>>>
106+
function(_Functor&& __f)
107+
//noexcept(_Handler<_Functor>::template _S_nothrow_init<_Functor>()) // CUDA does not support this line
108+
: _Function_base()
109+
110+
```
111+
Finally, build `opm-upscaling`:
112+
```
113+
git clone https://github.com/OPM/opm-upscaling.git
114+
mkdir opm-upscaling/build && cd opm-upscaling/build
115+
cmake ..
116+
make
117+
```
118+
119+
## Running OPM Flow on GPU
120+
121+
Running the Flow simulator on one or more GPUs requires to use GPU-specific solver and preconditionners. They can be defined in a linear solver [file](/assets/scripts/gpu-solver.json) in the JSON format, for example:
122+
```
123+
{
124+
"tol": "0.01",
125+
"maxiter": "200",
126+
"verbosity": "0",
127+
"solver": "gpubicgstab",
128+
"preconditioner": {
129+
"type": "GPUDILU",
130+
"verbosity" : 0,
131+
"split_matrix": "true",
132+
"tune_gpu_kernels": "true",
133+
"mixed_precision_scheme": 1
134+
}
135+
}
136+
```
137+
To run the simulation, open datasets are available [here](git clone https://github.com/OPM/opm-data.git). For example, to run the Norne use case, simply execute:
138+
```
139+
/home/ubuntu/opm-simulators/build/bin/flow NORNE_ATW2013.DATA --output-dir=out_gpu --matrix-add-well-contributions=true --linear-solver=/home/ubuntu/gpu-solver.json
140+
```
141+
for 1 GPU and:
142+
```
143+
mpirun -np N /home/ubuntu/opm-simulators/build/bin/flow NORNE_ATW2013.DATA --output-dir=out_gpu --matrix-add-well-contributions=true --threads-per-process=1 --linear-solver=/home/ubuntu/gpu-solver.json
144+
```
145+
for N GPUs.
146+
147+
## Notes:
148+
149+
Here are a few comments to take into account when considering running Flow on GPUs:
150+
* The options `--matrix-add-well-contributions=true` and `--threads-per-process=1` are recommended by OPM.
151+
* Because of the import CPU/GPU traffic, running Flow on GPU is only relevant for models above a certain size (few hundreds of thousands of cells).
152+
* Using a CUDA-aware version of Open MPI may improve the overall performance of the simulation.
153+
154+
## External Links
155+
156+
* [The Open Porous Media Initiative](https://opm-project.org/)
157+
* [Building CUDA-aware Open MPI](https://www.open-mpi.org/faq/?category=buildcuda)

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/OPM-Flow.png

413 KB
Loading

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/SLEIPNER_ORG_3D_View_SGAS_170_01_Dec_2010_FILTER.png

606 KB
Loading

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/build_flow_gpu.sh

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#!/bin/bash
2+
3+
for repo in opm-common opm-grid
4+
do
5+
git clone https://github.com/OPM/$repo.git
6+
mkdir $repo/build && cd $repo/build
7+
cmake ..
8+
make -j 8
9+
cd ../..
10+
done
11+
12+
git clone https://github.com/OPM/opm-simulators.git
13+
mkdir opm-simulators/build && cd opm-simulators/build
14+
cmake .. -DUSE_GPU_BRIDGE=OFF
15+
make -j 8
16+
cd ../..
17+
18+
git clone https://github.com/OPM/opm-upscaling.git
19+
mkdir opm-upscaling/build && cd opm-upscaling/build
20+
cmake ..
21+
make -j 8
22+
cd ../..
23+
24+
25+
26+
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
27+
435 | function(_Functor&& __f)
28+
| ^
29+
/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’
30+
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
31+
530 | operator=(_Functor&& __f)
32+
| ^
33+
/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’
34+
make[2]: *** [CMakeFiles/opmsimulators.dir/build.make:2680: CMakeFiles/opmsimulators.dir/opm/simulators/linalg/gpuistl/detail/vector_operations.cu.o] Error 1
35+
make[2]: *** Waiting for unfinished jobs....
36+
make[1]: *** [CMakeFiles/Makefile2:530: CMakeFiles/opmsimulators.dir/all] Error 2
37+
make: *** [Makefile:146: all] Error 2

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/gpu-solver.json

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"tol": "0.01",
3+
"maxiter": "200",
4+
"verbosity": "0",
5+
"solver": "gpubicgstab",
6+
"preconditioner": {
7+
"type": "GPUDILU",
8+
"verbosity" : 0,
9+
"split_matrix": "true",
10+
"tune_gpu_kernels": "true",
11+
"mixed_precision_scheme": 1
12+
}
13+
}

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/install_prerequisites.sh

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/bin/bash
2+
3+
# Make sure we have updated URLs to packages etc.
4+
sudo apt-get update -y
5+
6+
# For server edition of Ubuntu add-apt-repository depends on
7+
sudo apt-get install -y software-properties-common
8+
9+
# Add PPA for OPM packages
10+
sudo add-apt-repository -y ppa:opm/ppa
11+
sudo apt-get update -y
12+
13+
# Packages necessary for building
14+
sudo apt-get install -y build-essential gfortran pkg-config cmake
15+
16+
# Packages necessary for documentation
17+
sudo apt-get install -y doxygen ghostscript texlive-latex-recommended gnuplot
18+
19+
# Packages necessary for version control
20+
sudo apt-get install -y git-core
21+
22+
# MPI for parallel programs
23+
sudo apt-get install -y mpi-default-dev
24+
25+
# Prerequisite libraries
26+
sudo apt-get install -y libblas-dev libboost-all-dev libsuitesparse-dev libtrilinos-zoltan-dev libfmt-dev libcjson-dev
27+
28+
# Parts of Dune needed
29+
sudo apt-get install -y libdune-common-dev libdune-geometry-dev libdune-istl-dev libdune-grid-dev

0 commit comments

Comments
 (0)