Line data Source code
1 0 : \cond NEVER
2 : Distributed under the MIT License.
3 : See LICENSE.txt for details.
4 : \endcond
5 : # Installation {#installation}
6 :
7 : \tableofcontents
8 :
9 : This page details the installation procedure for SpECTRE on personal computers
10 : using x86-64 processors. For instructions on installing SpECTRE on clusters
11 : please refer to the \subpage installation_on_clusters page. For instructions on
12 : installing SpECTRE on Apple Silicon Macs, please refer to the
13 : \subpage installation_on_apple_silicon page. Refer to the
14 : \subpage versioning_and_releases page for information on specific versions
15 : to install.
16 :
17 : ### Install a specific version with Spack
18 :
19 : You can install SpECTRE with the [Spack](https://github.com/spack/spack) package
20 : manager:
21 :
22 : ```sh
23 : git clone https://github.com/spack/spack
24 : source ./spack/share/spack/setup-env.sh
25 : spack compiler find
26 : spack external find
27 : spack install spectre executables=ExportCoordinates3D \
28 : ^charmpp backend=multicore
29 : ```
30 :
31 : You probably want to customize your installation, e.g., to select a particular
32 : version of SpECTRE, the executables you want to install, additional options such
33 : as Python bindings, or the Charm++ backend. You can display all possible options
34 : with:
35 :
36 : ```sh
37 : spack info spectre # or charmpp, etc.
38 : ```
39 :
40 : Refer to the [Spack documentation](https://spack.readthedocs.io/en/latest/) for
41 : more information.
42 :
43 : ### Quick-start guide for code development with Docker and Visual Studio Code
44 :
45 : If you're new to writing code for SpECTRE and would like to jump right into a
46 : working development environment, a good place to start is our
47 : \subpage dev_guide_quick_start_docker_vscode. If you prefer setting up your
48 : development environment differently, read on!
49 :
50 : ### Dependencies
51 :
52 : **Note**: You don't need to install any of these dependencies by hand,
53 : or by using yum, apt, or other package managers; it is much easier
54 : to instead use Singularity, Docker, or Spack (see the corresponding
55 : sections below) to obtain an environment that includes
56 : all of these dependencies.
57 :
58 : #### Required:
59 :
60 : * [GCC](https://gcc.gnu.org/) 9.1 or later,
61 : [Clang](https://clang.llvm.org/) 13.0 or later (see
62 : [here](https://apt.llvm.org/) for how to get newer versions of clang through
63 : apt), or AppleClang 13.0.0 or later
64 : * [CMake](https://cmake.org/) 3.18.0 or later
65 : * [Charm++](http://charm.cs.illinois.edu/) 7.0.0, or later (experimental)
66 : \cite Charmpp1 \cite Charmpp2 \cite Charmpp3
67 : * [Git](https://git-scm.com/)
68 : * BLAS (e.g. [OpenBLAS](http://www.openblas.net))
69 : * [Blaze](https://bitbucket.org/blaze-lib/blaze/overview) v3.8. It can be
70 : beneficial to install Blaze with CMake so some configuration options are
71 : determined automatically, such as cache sizes. \cite Blaze1 \cite Blaze2
72 : * [Boost](http://www.boost.org/) 1.60.0 or later
73 : * [Brigand](https://github.com/edouarda/brigand) at commit
74 : 1c398e4f1e817ab195e4cd6fbb03c18cb386eea3 (late 2020) or later
75 : * [Catch2](https://github.com/catchorg/Catch2) 3.4.0 or later. Install from your
76 : package manager or do a standard CMake build and installation (as detailed
77 : in the [Catch2 docs](https://github.com/catchorg/Catch2/blob/devel/docs/cmake-integration.md#installing-catch2-from-git-repository)).
78 : Compile with `CMAKE_POSITION_INDEPENDENT_CODE=ON`.
79 : * [GSL](https://www.gnu.org/software/gsl/) \cite Gsl
80 : * [HDF5](https://support.hdfgroup.org/HDF5/) (non-mpi version on macOS)
81 : \cite Hdf5
82 : * [jemalloc](https://github.com/jemalloc/jemalloc)
83 : * LAPACK
84 : * [libsharp](https://github.com/Libsharp/libsharp) should be built with
85 : support disabled for openmp and mpi, as we want all of our parallelism to
86 : be accomplished via Charm++. \cite Libsharp
87 : * [LIBXSMM](https://github.com/hfp/libxsmm) version 1.16.1 or later
88 : \cite Libxsmm
89 : * [yaml-cpp](https://github.com/jbeder/yaml-cpp) version 0.6.3 or later.
90 : Building with shared library support is also recommended. \cite Yamlcpp
91 : * [Python](https://www.python.org/) 3.8 or later.
92 : * Python dependencies listed in `support/Python/requirements.txt`.
93 : Install with `pip3 install -r support/Python/requirements.txt`.
94 : Make sure you are working in a [Python venv](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment)
95 : before installing packages.
96 : Alternatively, you can set `BOOTSTRAP_PY_DEPS=ON` when configuring a build
97 : with CMake to install missing Python packages into the build directory
98 : automatically.
99 : <details>
100 : \include support/Python/requirements.txt
101 : </details>
102 :
103 : #### Optional:
104 : * [Pybind11](https://pybind11.readthedocs.io) 2.6.0 or later for SpECTRE Python
105 : bindings \cite Pybind11
106 : * [Doxygen](https://www.doxygen.nl/index.html) 1.9.1 to 1.9.6 — to
107 : generate documentation
108 : * Python dev dependencies listed in `support/Python/dev_requirements.txt`
109 : — for documentation pre- and post-processing, formatting code, etc.
110 : Install with `pip3 install -r support/Python/dev_requirements.txt`.
111 : Make sure you are working in a [Python venv](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment)
112 : before installing packages.
113 : <details>
114 : \include support/Python/dev_requirements.txt
115 : </details>
116 : * [Google Benchmark](https://github.com/google/benchmark) - to do
117 : microbenchmarking inside the SpECTRE framework. v1.2 or newer is required
118 : * [LCOV](http://ltp.sourceforge.net/coverage/lcov.php) and
119 : [gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html) — to check code test
120 : coverage
121 : * [PAPI](http://icl.utk.edu/papi/) — to access hardware performance counters
122 : * [ClangFormat](https://clang.llvm.org/docs/ClangFormat.html) — to format C++
123 : code in a clear and consistent fashion
124 : * [Clang-Tidy](http://clang.llvm.org/extra/clang-tidy/) — to "lint" C++ code
125 : * [Scotch](https://gitlab.inria.fr/scotch/scotch) - to build the `ScotchLB`
126 : graph partition based load balancer in charm++.
127 : * [ffmpeg](https://www.ffmpeg.org/) - for animating 1d simulations with
128 : matplotlib
129 : * [xsimd](https://github.com/xtensor-stack/xsimd) 11.0.1 or newer - for manual
130 : vectorization
131 : * [libbacktrace](https://github.com/ianlancetaylor/libbacktrace) - to show
132 : source files and line numbers in backtraces of errors and asserts. Available
133 : by default on many systems, so you may not have to install it at all. The
134 : CMake configuration will tell you if you have libbacktrace installed.
135 : * [ParaView](https://www.paraview.org/) - for visualization \cite Paraview1
136 : \cite Paraview2 . Make sure your ParaView installation uses the same (major
137 : and minor) version of Python as the rest of the build.
138 : * [SpEC](https://www.black-holes.org/code/SpEC.html) - to load SpEC data.
139 : Compile the exporter in SpEC's `Support/ApplyObservers/Exporter/` directory
140 : (see the `Makefile` in that directory). Also make sure to compile SpEC with
141 : the same compiler and MPI as SpECTRE to avoid compatibility issues.
142 :
143 : ## Clone the SpECTRE repository
144 :
145 : First, clone the [SpECTRE repository](https://github.com/sxs-collaboration/spectre)
146 : to a directory of your choice. In the following we will refer to it as
147 : SPECTRE_ROOT. You may `git clone` from GitHub, in which case SPECTRE_ROOT will
148 : be `<your_current_directory>/spectre`. That is, inside SPECTRE_ROOT are `docs`,
149 : `src`, `support`, `tests` etc. You can also download the source and extract them
150 : to your desired working directory, making sure not to leave out hidden files
151 : when you `cp` or `mv` the source files.
152 :
153 : ## Using Docker to obtain a SpECTRE environment {#docker_install}
154 :
155 : A [Docker](https://www.docker.com/) image is available from
156 : [DockerHub](https://hub.docker.com/r/sxscollaboration/spectre/) and can
157 : be used to build SpECTRE on a personal machine.
158 :
159 : **Note**: The Docker image or the Singularity image (see below) are
160 : the recommended ways of using SpECTRE on a personal
161 : Linux machine. Because of the wide variety of operating systems available today
162 : it is not possible for us to support all configurations. However, using Spack
163 : as outlined below is a supported alternative to Docker or Singularity images.
164 :
165 : **Note**: If you have SELinux active
166 : on your system you must figure out how to enable sharing files with the host
167 : OS. If you receive errors that you do not have permission to access a shared
168 : directory it is likely that your system has SELinux enabled. One option is to
169 : disable SELinux at the expense of reducing the security of your system.
170 :
171 : To build with the Docker image:
172 :
173 : 1. Install [Docker-Desktop](https://docs.docker.com/get-docker/). For Linux, if
174 : you want to be able to run the following steps without `sudo`, follow the
175 : [post-installation-guide](https://docs.docker.com/engine/install/linux-postinstall/)
176 : to add a non-root user.
177 :
178 : 2. Retrieve the Docker image (you may need `sudo` in front of this command)
179 : ```
180 : docker pull sxscollaboration/spectre:dev
181 : ```
182 : 3. Start the Docker container (you may need `sudo`)
183 : ```
184 : docker run -v $SPECTRE_ROOT/:$SPECTRE_ROOT/ --name spectre_dev \
185 : -i -t sxscollaboration/spectre:dev /bin/bash
186 : ```
187 : - `-v $SPECTRE_ROOT/:$SPECTRE_ROOT/` binds the directory `$SPECTRE_ROOT`
188 : (which is an environment variable you must set up or just use the actual
189 : path) outside the container to `$SPECTRE_ROOT` inside the container. In this
190 : way, files in the `$SPECTRE_ROOT` on your host system (outside the container)
191 : become accessible within the container through the directory SPECTRE_ROOT
192 : inside the container. If you wonder why the same SPECTRE_ROOT needs to be
193 : used for both inside and outside the container, which is why `$SPECTRE_ROOT`
194 : is repeated in the command above with separated by a colon, please see one of
195 : the notes below regarding `-v` flag.
196 : - The `--name spectre_dev` is optional. If you don't name your container,
197 : docker will generate an arbitrary name.
198 : - On macOS you can significantly increase the performance of file system
199 : operations by appending the flag `:delegated` to `-v`, e.g.
200 : `-v $SPECTRE_ROOT/:$SPECTRE_ROOT/:delegated` (see
201 : https://docs.docker.com/docker-for-mac/osxfs-caching/).
202 : - The `-i` flag is for interactive mode, which will drop you into the
203 : container.
204 : - It can be useful to expose a port to the host so you can run servers such
205 : as [Jupyter](https://jupyter.org/index.html) for accessing the Python
206 : bindings (see \ref spectre_using_python) or a Python web server to view the
207 : documentation. To do so, append the `-p` option, e.g. `-p 8000:8000`.
208 :
209 : You will end up in a bash shell in the docker container,
210 : as root (you need to be root).
211 : Within the container, the files in `$SPECTRE_ROOT` are available and Charm++
212 : is installed in `/work/charm_7_0_0`. For the following steps, stay inside the
213 : docker container as root.
214 : 4. Proceed with [building SpECTRE](#building-spectre).
215 :
216 : **Notes:**
217 : * Everything in your build directory is owned by root, and is
218 : accessible only within the container.
219 : * You should edit source files in SPECTRE_ROOT in a separate terminal
220 : outside the container, and use the container only for compiling and
221 : running the code.
222 : * If you exit the container (e.g. ctrl-d),
223 : your compilation directories are still saved, as are any other changes to
224 : the container that you have made.
225 : To restart the container, try the following commands
226 : (you may need `sudo`):
227 : 1. `docker ps -a`,
228 : to list all containers with their CONTAINER_IDs and CONTAINER_NAMEs,
229 : 2. `docker start -i CONTAINER_NAME` or `docker start -i CONTAINER_ID`,
230 : to restart your container (above, the CONTAINER_NAME was spectre_dev).
231 : * When the Docker container gets updated, you can stop it with
232 : `docker stop CONTAINER_NAME`, remove it with `docker rm CONTAINER_NAME`
233 : and then start at step 2 above to run it again.
234 : * You can run more than one shell in the same container, for instance
235 : one shell for compiling with gcc and another for compiling
236 : with clang.
237 : To add a new shell, run `docker exec -it CONTAINER_NAME /bin/bash`
238 : (or `docker exec -it CONTAINER_ID /bin/bash`) from
239 : a terminal outside the container.
240 : * In step 4 above, technically docker allows you to say
241 : `-v $SPECTRE_ROOT/:/my/new/path` to map `$SPECTRE_ROOT` outside the
242 : container to any path you want inside the container, but **do not do this**.
243 : Compiling inside the container sets up git hooks in SPECTRE_ROOT that
244 : contain hardcoded pathnames to SPECTRE_ROOT *as seen from inside the
245 : container*. So if your source paths inside and outside the container are
246 : different, commands like `git commit` run *from outside the container* will
247 : die with `No such file or directory`.
248 : * If you want to use Docker within VSCode, take a look at our
249 : [quick start guide](../DevGuide/QuickStartDockerVSCode.md) for using Docker
250 : with VSCode.
251 :
252 : ## Using Singularity to obtain a SpECTRE environment
253 :
254 : [Singularity](https://sylabs.io) is a container alternative
255 : to Docker with better security and nicer integration.
256 :
257 : To build SpECTRE with Singularity you must:
258 :
259 : 1. Build [Singularity](https://sylabs.io) and add it to your
260 : `$PATH`
261 : 2. `cd` to the directory where you want to store the SpECTRE Singularity image,
262 : source, and build directories, let's call it WORKDIR. The WORKDIR must be
263 : somewhere in your home directory. If this does not work for you, follow the
264 : Singularity instructions on setting up additional [bind
265 : points](https://sylabs.io/guides/3.7/user-guide/bind_paths_and_mounts.html)
266 : (version 3.7. For other versions, see the [docs](https://sylabs.io/docs/)).
267 : Once inside the WORKDIR, clone SpECTRE into `WORKDIR/SPECTRE_ROOT`.
268 : 3. Run `sudo singularity build spectre.img
269 : docker://sxscollaboration/spectre:dev`.
270 : You can also use spectre:ci instead of spectre:dev if you want more
271 : compilers installed.
272 :
273 : If you get the error message that `makesquashfs` did not have enough space to
274 : create the image you need to set a different `SINGULARITY_TMPDIR`. This can
275 : be done by running: `sudo SINGULARITY_TMPDIR=/path/to/new/tmp singularity
276 : build spectre.img docker://sxscollaboration/spectre:dev`. Normally
277 : `SINGULARITY_TMPDIR` is `/tmp`, but building the image will temporarily need
278 : almost 8GB of space.
279 :
280 : You can control where Singularity stores the downloaded image files from
281 : DockerHub by specifying the `SINGULARITY_CACHEDIR` environment variable. The
282 : default is `$HOME/.singularity/`. Note that `$HOME` is `/root` when running
283 : using `sudo`.
284 : 4. To start the container run `singularity shell spectre.img` and you
285 : will be dropped into a bash shell.
286 : 5. Proceed with [building SpECTRE](#building-spectre).
287 :
288 : **Notes:**
289 : - You should edit source files in SPECTRE_ROOT in a separate terminal
290 : outside the container, and use the container only for compiling and running
291 : the code.
292 : - If you don't have the same Python version in your environment outside the
293 : container as the version inside the container, this will create problems
294 : with git hooks. The Singularity container uses python3.8 by default. Thus, it
295 : is up to the user to ensure that they are using the same Python version inside
296 : and outside the container. To use a different Python version in the container
297 : add `-D Python_EXECUTABLE=/path/to/python` to the cmake command where
298 : `/path/to/python` is usually `/usr/bin/pythonX` and `X` is the version you
299 : want.
300 : - Unlike Docker, Singularity does not keep the state between runs. However, it
301 : shares the home directory with the host OS so you should do all your work
302 : somewhere in your home directory.
303 : - To run more than one container just do `singularity shell spectre.img` in
304 : another terminal.
305 : - Since the data you modify lives on the host OS there is no need to worry about
306 : losing any data, needing to clean up old containers, or sharing data between
307 : containers and the host.
308 :
309 : ## Using Spack to set up a SpECTRE environment
310 :
311 : SpECTRE's dependencies can be installed with
312 : [Spack](https://github.com/spack/spack), a package manager tailored for HPC use.
313 : [Install Spack](https://spack.readthedocs.io/en/latest/getting_started.html) by
314 : cloning it into `SPACK_DIR` (a directory of your choice). Then, enable Spack's
315 : shell support with `source SPACK_DIR/share/spack/setup-env.sh`. Consider adding
316 : this line to your `.bash_profile`, `.bashrc`, or similar. Refer to [Spack's
317 : getting started guide](https://spack.readthedocs.io/en/latest/getting_started.html)
318 : for more information.
319 :
320 : Once you have Spack installed, one way to install the SpECTRE dependencies is
321 : with a [Spack environment](https://spack.readthedocs.io/en/latest/environments.html):
322 :
323 : \include support/DevEnvironments/spack.yaml
324 :
325 : You can also install the Spack packages listed in the environment file above
326 : with a plain `spack install` if you prefer.
327 :
328 : **Notes:**
329 : - Spack allows very flexible configurations and we recommended you read the
330 : [documentation](https://spack.readthedocs.io) if you require features such as
331 : packages installed with different compilers.
332 : - For security, it is good practice to make Spack [use the system's
333 : OpenSSL](https://spack.readthedocs.io/en/latest/getting_started.html#openssl)
334 : rather than allow it to install a new copy.
335 : - To avoid reinstalling lots of system-provided packages with Spack, use the
336 : `spack external find` feature and the `--reuse` flag to `spack concretize` (or
337 : `spack install`). You can also install some of the dependencies with your
338 : system's package manager in advance, e.g., with `apt` or `brew`. If they are
339 : not picked up by `spack external find` automatically, register them with Spack
340 : manually. See the [Spack documentation on external
341 : packages](https://spack.readthedocs.io/en/latest/build_settings.html#external-packages)
342 : for details.
343 : - Spack works well with a module environment, such as
344 : [LMod](https://github.com/TACC/Lmod). See the [Spack documentation on
345 : modules](https://spack.readthedocs.io/en/latest/module_file_support.html) for
346 : details.
347 : - On macOS:
348 : - Brigand has an issue with AppleClang 13 when compiling tests (see
349 : https://github.com/edouarda/brigand/issues/274). Since it is header-only,
350 : you can simply clone the [Brigand repository](https://github.com/edouarda/brigand)
351 : instead of installing it with Spack.
352 :
353 : ## Building Charm++ {#building-charm}
354 :
355 : If you are not using a container, haven't installed Charm++ with Spack, or want
356 : to install Charm++ manually for other reasons, follow the installation
357 : instructions in the [Charm++ repository](https://github.com/UIUC-PPL/charm)
358 : and in their [documentation](https://charm.readthedocs.io/en/latest/quickstart.html#installing-charm).
359 : Here are a few notes:
360 :
361 : - Once you cloned the [Charm++ repository](https://github.com/UIUC-PPL/charm),
362 : run `git checkout v7.0.0` to switch to a supported, stable release of
363 : Charm++.
364 : - Apply the appropriate patch (if there is one) for the version from
365 : `${SPECTRE_ROOT}/support/Charm`. For example, if you have Charm++ v7.0.0
366 : then the patch will be `v7.0.0.patch`.
367 : - Choose the `LIBS` target to compile. This is needed so that we can support the
368 : more sophisticated load balancers in SpECTRE executables.
369 : - On a personal machine the correct target architecture is likely
370 : `multicore-linux-x86_64`, or `multicore-darwin-x86_64` on macOS. On an HPC
371 : system the correct Charm++ target architecture depends on the machine's
372 : inter-node communication architecture. It might take some experimenting to
373 : figure out which Charm++ configuration provides the best performance.
374 : - Compile Charm++ with support for shared libraries by appending the option
375 : `--build-shared` to the `./build` command or pass `BUILD_SHARED=ON` to the
376 : CMake configuration (see the [Charm++ installation
377 : instructions](https://github.com/UIUC-PPL/charm#building-dynamic-libraries)).
378 : - When compiling Charm++ you can specify the compiler using, for example,
379 : ```
380 : ./build LIBS ARCH clang
381 : ```
382 :
383 : ## Building SpECTRE {#building-spectre}
384 :
385 : Once you have set up your development environment you can compile SpECTRE.
386 : Follow these steps:
387 :
388 : 1. Create a build directory where you would like to compile SpECTRE. In the
389 : Docker container you could create, e.g., `/work/spectre-build`. It can be
390 : useful to add a descriptive label to the name of the build directory since
391 : you may create more later, e.g., `build-clang-Debug`. Then, `cd` into the
392 : build directory.
393 : 2. Determine the location of your Charm++ installation. In the Docker container
394 : it is `/work/charm_7_0_0/multicore-linux-x86_64-gcc` for GCC builds and
395 : `/work/charm_7_0_0/mpi-linux-x86_64-smp-clang` for clang builds. For Spack
396 : installations you can determine it with
397 : `spack location --install-dir charmpp`. We refer to the install directory as
398 : `CHARM_ROOT` below.
399 : 3. In your new SpECTRE build directory, configure the build with CMake:
400 : ```
401 : cmake -D CHARM_ROOT=$CHARM_ROOT SPECTRE_ROOT
402 : ```
403 : Add options to the `cmake` command to to configure the build, select
404 : compilers, etc. For instance, to build with clang you may run:
405 : ```
406 : cmake -D CMAKE_CXX_COMPILER=clang++ \
407 : -D CMAKE_C_COMPILER=clang \
408 : -D CMAKE_Fortran_COMPILER=gfortran \
409 : -D CHARM_ROOT=$CHARM_ROOT \
410 : SPECTRE_ROOT
411 : ```
412 : See \ref common_cmake_flags for documentation on possible configuration
413 : options.
414 : 4. When cmake configuration is done, you are ready to build target executables.
415 : - You can see the list of available targets by running `make list` (or `ninja
416 : list` if you are using the Ninja generator) or by using tab completion.
417 : Compile targets with `make -jN TARGET` (or `ninja -jN TARGET`), where `N`
418 : is the number of cores to build on in parallel (e.g. `-j4`). Note that the
419 : Ninja generator allows you to compile individual source files too.
420 : - Compile the `unit-tests` target and run `ctest -L unit` to run unit tests.
421 : Compile `test-executables` and run `ctest` to run all tests, including
422 : executables. To compile `test-executables` you may have to reduce the
423 : number of cores you build on in parallel to avoid running out of memory.
424 : - To use the command-line interface (CLI), compile the `cli` target (see
425 : \ref tutorial_cli).
426 : - To use the Python bindings, compile the `all-pybindings` target (see
427 : \ref spectre_using_python).
428 :
429 : ## Code Coverage Analysis
430 :
431 : For any coverage analysis you will need to have LCOV installed on the system.
432 : For documentation coverage analysis you will also need to install
433 : [coverxygen](https://github.com/psycofdj/coverxygen) and for test coverage
434 : analysis [gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html).
435 :
436 : If you have these installed (which is already done if
437 : you are using the docker container), you can look at code coverage as follows:
438 :
439 : 1. On a gcc build, pass `-D COVERAGE=ON` to `cmake`
440 : 2. `make unit-test-coverage`
441 : 3. The output is in `docs/html/unit-test-coverage`.
|
