Skip to content

Commit 15ea342

Browse files
bendudsonbendudson
authored
Merge pull request #298 from boutproject/install-docs
Expanded compilation docs
2 parents d9546c6 + e04f571 commit 15ea342

File tree

3 files changed

+248
-104
lines changed

3 files changed

+248
-104
lines changed

‎docs/sphinx/getting_started.rst

-103
This file was deleted.

‎docs/sphinx/index.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Welcome to Hermes-3 documentation!
88
:name: contents
99

1010
introduction
11-
getting_started
11+
installation_using_cmake
1212
installation_using_spack
1313
examples
1414
tests

‎docs/sphinx/installation_using_cmake.rst

+247
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
.. _sec-installation_using_cmake:
2+
3+
Installation using CMake
4+
===============
5+
6+
Compilation process
7+
----------
8+
9+
Compilation is achieved in two stages - the first is configuration where all the compile-time
10+
options are read in. The second is the build which results in a ready-to-use Hermes-3 installation
11+
in a directory named `build` by default. The build directory name can be changed to have
12+
multiple builds available at the same time.
13+
14+
If you make changes to the code, you can skip straight to the build stage to save time.
15+
Only modified files will be recompiled.
16+
17+
Hermes-3 is built using `CMake <https://cmake.org>`_. During configuration `BOUT++
18+
<https://github.com/boutproject/BOUT-dev/>`_ will be automatically
19+
downloaded as a submodule, together with some dependencies. The correct version
20+
of `netCDF <https://www.unidata.ucar.edu/software/netcdf/>`_ is downloaded
21+
and compiled automatically for convenience. `FFTW
22+
<https://www.fftw.org/>`_ is assumed to be installed already.
23+
24+
Hermes-3 uses two solvers: `SUNDIALS <https://computing.llnl.gov/projects/sundials>`_ `cvode` for
25+
time-dependent simulations and the faster `PETSc
26+
<https://petsc.org>`_ `beuler` for steady-state transport problems. While SUNDIALS
27+
can be downloaded and installed automatically, PETSc requires manual installation.
28+
29+
Installing with SUNDIALS (cvode) only
30+
----------
31+
32+
If you only want to use the `cvode` solver, then the
33+
recommended way to build Hermes-3 links to the SUNDIALS library:
34+
35+
36+
1. Configure with cmake, downloading and linking to SUNDIALS and NetCDF4:
37+
38+
.. code-block:: bash
39+
40+
cmake . -B build -DBOUT_DOWNLOAD_SUNDIALS=ON -DBOUT_DOWNLOAD_NETCDF_CXX4=ON
41+
42+
2. Build, compiling Hermes-3 and all dependencies using 4 parallel cores
43+
(adjust as necessary):
44+
45+
.. code-block:: bash
46+
47+
cmake --build build -j 4
48+
49+
3. Run the unit and integrated tests to check that everything is working:
50+
51+
.. code-block:: bash
52+
53+
cd build
54+
ctest
55+
56+
Note that the integrated tests require MPI, and so may not run on the
57+
head nodes of many computing clusters.
58+
59+
60+
Installing with SUNDIALS and PETSc (beuler)
61+
-------------------
62+
63+
The steady-state solver beuler requires PETSc and is often preconditioned using the `hypre`
64+
package, which is automatically downloaded and configured during PETSc installation.
65+
66+
Here is an example PETSc configure setup:
67+
68+
.. code-block:: bash
69+
70+
./configure --with-mpi=yes --download-hypre --download-make --with-fortran-bindings=0 --with-debugging=0
71+
72+
Here is an example working script to automatically download and compile PETSc on `Viking2`:
73+
74+
.. code-block:: bash
75+
76+
mkdir petsc-build
77+
wget https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz
78+
tar xzf petsc-3.17.4.tar.gz
79+
cd petsc-3.17.4
80+
./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" --download-hypre --with-debugging=0 --prefix=../petsc-build
81+
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt all
82+
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt install
83+
make -j 4 PETSC_DIR=$PWD/../petsc-build PETSC_ARCH="" check
84+
85+
And one for `ARCHER2`:
86+
87+
.. code-block:: bash
88+
89+
mkdir petsc-build
90+
wget https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz
91+
tar xzf petsc-3.17.4.tar.gz
92+
cd petsc-3.17.4
93+
./configure --CC=cc --CXX=CC --FC=ftn COPTFLAGS="-Ofast" CXXOPTFLAGS="-Ofast" FOPTFLAGS="-Ofast" --with-batch --known-64-bit-blas-indices=0 --known-sdor-returns-double=0 --known-snrm2-returns-double=0 --with-fortran-bindings=0 --download-hypre --with-debugging=0 --prefix=../petsc-build
94+
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt all
95+
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt install
96+
make -j 4 PETSC_DIR=$PWD/../petsc-build PETSC_ARCH="" check
97+
98+
Once PETSc is installed, link it to Hermes-3 using the ``-DBOUT_USE_PETSC=ON`` CMake flag:
99+
100+
.. code-block:: bash
101+
102+
cmake . -B build -DBOUT_DOWNLOAD_SUNDIALS=ON -DBOUT_DOWNLOAD_NETCDF_CXX4=ON -DBOUT_USE_PETSC=ON
103+
104+
If the ``PETSC_DIR`` and ``PETSC_ARCH`` environment variables have been set,
105+
then CMake should pick them up. If it doesn't, try doing a clean build by removing
106+
any previously generated build directories.
107+
108+
109+
Dependencies
110+
-------------------
111+
Since Hermes-3 heavily relies on BOUT++, the `BOUT++ documentation on installation and
112+
dependencies <https://bout-dev.readthedocs.io/en/stable/user_docs/quickstart.html#prerequisites>`_
113+
contains a lot of useful information. Below is a selection of working module lists
114+
for several HPC systems:
115+
116+
YPI Workstations:
117+
118+
.. code-block:: bash
119+
120+
module load mpi/OpenMPI/4.1.1-GCC-10.3.0
121+
module load devel/CMake/3.20.1-GCCcore-10.3.0
122+
module load numlib/OpenBLAS/0.3.15-GCC-10.3.0
123+
module load lib/FlexiBLAS/3.0.4-GCC-10.3.0
124+
125+
ARCHER2:
126+
127+
.. code-block:: bash
128+
129+
module swap PrgEnv-cray/8.3.3
130+
module swap cce/15.0.0
131+
module swap cray-mpich/8.1.23
132+
module load cray-python/3.9.13.1
133+
module load netcdf4
134+
module load cmake
135+
module load cray-hdf5
136+
module load cray-netcdf/4.9.0.1
137+
module load cray-parallel-netcdf/1.12.3.1
138+
module load cray-fftw/3.3.10.3
139+
module load valgrind4hpc
140+
141+
Marconi:
142+
143+
.. code-block:: bash
144+
145+
module load tools/git/2.32.0-GCCcore-10.3.0-nodocs
146+
module load mpi/OpenMPI/4.1.1-GCC-10.3.0
147+
module load devel/CMake/3.20.1-GCCcore-10.3.0
148+
module load numlib/OpenBLAS/0.3.15-GCC-10.3.0
149+
module load data/netCDF/4.8.0-gompi-2021a
150+
module load lang/SciPy-bundle/2021.05-foss-2021a
151+
152+
Viking2:
153+
154+
.. code-block:: bash
155+
156+
module load OpenMPI/4.1.1-GCC-10.3.0
157+
module load git/2.32.0-GCCcore-10.3.0-nodocs
158+
module load CMake/3.20.1-GCCcore-10.3.0
159+
module load OpenBLAS/0.3.15-GCC-10.3.0
160+
module load netCDF/4.8.0-gompi-2021a
161+
module load SciPy-bundle/2021.05-foss-2021a
162+
163+
Ancalagon:
164+
165+
.. code-block:: bash
166+
167+
module load OpenMPI/4.1.1-GCC-10.3.0
168+
module load CMake/3.20.1-GCCcore-10.3.0
169+
module load OpenBLAS/0.3.15-GCC-10.3.0
170+
module load SciPy-bundle/2021.05-foss-2021a
171+
module load netCDF/4.8.0-gompi-2021a
172+
173+
174+
Slope (flux) limiter settings
175+
-----------------
176+
177+
Advection operators in Hermes-3 use slope limiters, also called `flux
178+
limiters <https://en.wikipedia.org/wiki/Flux_limiter>`_ to suppress
179+
spurious numerical oscillations near sharp features, while converging
180+
at 2nd-order in smooth regions. In general there is a trade-off
181+
between suppression of numerical oscillations and dissipation: Too
182+
little dissipation results in oscillations that can cause problems
183+
(e.g. negative densities), while too much dissipation smooths out real
184+
features and requires higher resolution to converge to the same
185+
accuracy. The optimal choice of method is problem-dependent.
186+
187+
The CMake flag ``-DHERMES_SLOPE_LIMITER`` sets the choice of slope
188+
limiter. The default method is ``MC``, which has been found to
189+
provide a good balance for problems of interest. If more dissipation
190+
is required then this can be changed to ``MinMod``;
191+
if less dissipation is required then this can be changed
192+
to ``Superbee``.
193+
194+
The appropriate limiter is problem-dependent. ``MinMod`` can work well
195+
for 1D tokamak simulations with steep gradients, e.g. simulations of detachment
196+
transients in high power machines which are already under-dissipative
197+
due to the lack of cross-field transport. The use of ``MinMod`` in 2D or 3D can
198+
lead to over-dissipation, but greater robustness.
199+
200+
201+
Compiling in debug mode
202+
-----------------
203+
Please see the `relevant page <https://bout-dev.readthedocs.io/en/stable/user_docs/advanced_install.html#optimisation-and-run-time-checking>`_
204+
in the BOUT++ documentation.
205+
206+
207+
Custom versions of BOUT++
208+
-----------------
209+
210+
If you have already installed BOUT++ and want to use that rather than
211+
configure and build BOUT++ again, set ```-HERMES_BUILD_BOUT=OFF``` and pass
212+
CMake the path to the BOUT++ `build` directory e.g.
213+
214+
.. code-block:: bash
215+
216+
cmake . -B build -DHERMES_BUILD_BOUT=OFF -DCMAKE_PREFIX_PATH=$HOME/BOUT-dev/build
217+
218+
The version of BOUT++ required by Hermes-3 is periodically updated, and is usually derived
219+
from a commit on the `next` branch of BOUT++. The up to date commit can be found in the
220+
`"external" directory of the Hermes-3 repo
221+
<https://github.com/bendudson/hermes-3/tree/master/external>`_.
222+
223+
224+
Custom configuration of CMake
225+
-----------------
226+
227+
The CMake configuration can be customised: See the `BOUT++
228+
documentation
229+
<https://bout-dev.readthedocs.io/en/latest/user_docs/installing.html#cmake>`_
230+
for examples of using `cmake` arguments, or edit the compile options
231+
interactively before building:
232+
233+
.. code-block:: bash
234+
235+
ccmake . -B build
236+
237+
238+
Troubleshooting issues
239+
-----------------
240+
241+
The first thing to try is to remove the build directory for a clean
242+
compilation. If you use Conda, it can cause several issues (e.g. making
243+
BOUT++ pick up the Conda MPI installation instead of the module one). A
244+
workaround is to compile with the CMake flag `-DBOUT_IGNORE_CONDA_ENV=ON`.
245+
246+
247+

0 commit comments

Comments
 (0)