SpECTRE
v2024.09.29
|
This page details how to install SpECTRE on personal machines and on clusters that have no official support (yet).
A quick way to run the code without installing anything at all is with our containerized releases:
You can also use Apptainer/Singularity instead of Docker, which works better on computing clusters and is more convenient because it shares the host's file system:
The entrypoint to this container is the SpECTRE command-line interface (CLI). For example, you can generate initial data for a simulation of merging black holes and plot the result like this:
The containers currently have precompiled code only for Linux x86_64 platforms, and only have a limited set of executables precompiled. The supported features available in the precompiled containers are:
Another way of running the code without installing anything is with our precompiled static binaries, which are published on GitHub:
These are currently compiled only for Linux x86_64 platforms and for Intel Haswell architecture, so they should be compatible with machines newer than mid 2013.
We only publish a limited set of precompiled static binaries that are useful as stand-alone tools, such as the CCE executables (see Running CCE).
If you're new to writing code for SpECTRE and would like to jump right into a working development environment, a good place to start is our Code development quick-start with Docker and Visual Studio Code.
The easiest way of installing SpECTRE natively on a new machine is this:
Clone the SpECTRE repository:
Install Charm++:
Choose the <version>
from this list in the Charm++ documentation. For example, choose multicore-linux-x86_64
on a Linux laptop, multicore-darwin-arm8
on an Apple Silicon laptop, and mpi-linux-x86_64
on a standard computing cluster (you will also need MPI for this). See Building Charm++ for details.
Configure and build SpECTRE:
See Building SpECTRE for details and Commonly Used CMake flags for a list of possible configuration options. For example, set -D ENABLE_OPENMP=ON
to enable OpenMP-parallelization for the exporter library.
Now you can compile executables (again, see Building SpECTRE for details). For example:
You can also install SpECTRE with the Spack package manager:
You probably want to customize your installation, e.g., to select a particular version of SpECTRE, the executables you want to install, additional options such as Python bindings, or the Charm++ backend. You can display all possible options with:
Refer to the Spack documentation for more information.
This remainder of this page details the installation procedure for SpECTRE.
The following dependencies will be fetched automatically if you set SPECTRE_FETCH_MISSING_DEPS=ON
:
CMAKE_POSITION_INDEPENDENT_CODE=ON
.support/Python/requirements.txt
. Install with pip3 install -r support/Python/requirements.txt
. Make sure you are working in a Python venv before installing packages. Alternatively, you can set BOOTSTRAP_PY_DEPS=ON
when configuring a build with CMake to install missing Python packages into the build directory automatically. support/Python/requirements.txt
. [97]support/Python/dev_requirements.txt
— for documentation pre- and post-processing, formatting code, etc. Install with pip3 install -r support/Python/dev_requirements.txt
. Make sure you are working in a Python venv before installing packages. ScotchLB
graph partition based load balancer in charm++.Support/ApplyObservers/Exporter/
directory (see the Makefile
in that directory). Also make sure to compile SpEC with the same compiler and MPI as SpECTRE to avoid compatibility issues.First, clone the SpECTRE repository to a directory of your choice. In the following we will refer to it as SPECTRE_ROOT. You may git clone
from GitHub, in which case SPECTRE_ROOT will be <your_current_directory>/spectre
. That is, inside SPECTRE_ROOT are docs
, src
, support
, tests
etc. You can also download the source and extract them to your desired working directory, making sure not to leave out hidden files when you cp
or mv
the source files.
A Docker image is available from DockerHub and can be used to build SpECTRE on a personal machine.
Note: If you have SELinux active on your system you must figure out how to enable sharing files with the host OS. If you receive errors that you do not have permission to access a shared directory it is likely that your system has SELinux enabled. One option is to disable SELinux at the expense of reducing the security of your system.
To build with the Docker image:
sudo
, follow the post-installation-guide to add a non-root user.sudo
in front of this command) Start the Docker container (you may need sudo
)
-v $SPECTRE_ROOT/:$SPECTRE_ROOT/
binds the directory $SPECTRE_ROOT
(which is an environment variable you must set up or just use the actual path) outside the container to $SPECTRE_ROOT
inside the container. In this way, files in the $SPECTRE_ROOT
on your host system (outside the container) become accessible within the container through the directory SPECTRE_ROOT inside the container. If you wonder why the same SPECTRE_ROOT needs to be used for both inside and outside the container, which is why $SPECTRE_ROOT
is repeated in the command above with separated by a colon, please see one of the notes below regarding -v
flag.--name spectre_dev
is optional. If you don't name your container, docker will generate an arbitrary name.:delegated
to -v
, e.g. -v $SPECTRE_ROOT/:$SPECTRE_ROOT/:delegated
(see https://docs.docker.com/docker-for-mac/osxfs-caching/).-i
flag is for interactive mode, which will drop you into the container.-p
option, e.g. -p 8000:8000
.You will end up in a bash shell in the docker container, as root (you need to be root). Within the container, the files in $SPECTRE_ROOT
are available and Charm++ is installed in /work/charm_7_0_0
. For the following steps, stay inside the docker container as root.
Notes:
sudo
):docker ps -a
, to list all containers with their CONTAINER_IDs and CONTAINER_NAMEs,docker start -i CONTAINER_NAME
or docker start -i CONTAINER_ID
, to restart your container (above, the CONTAINER_NAME was spectre_dev).docker stop CONTAINER_NAME
, remove it with docker rm CONTAINER_NAME
and then start at step 2 above to run it again.docker exec -it CONTAINER_NAME /bin/bash
(or docker exec -it CONTAINER_ID /bin/bash
) from a terminal outside the container.-v $SPECTRE_ROOT/:/my/new/path
to map $SPECTRE_ROOT
outside the container to any path you want inside the container, but do not do this. Compiling inside the container sets up git hooks in SPECTRE_ROOT that contain hardcoded pathnames to SPECTRE_ROOT as seen from inside the container. So if your source paths inside and outside the container are different, commands like git commit
run from outside the container will die with No such file or directory
.Singularity is a container alternative to Docker with better security and nicer integration.
To build SpECTRE with Singularity you must:
$PATH
cd
to the directory where you want to store the SpECTRE Singularity image, source, and build directories, let's call it WORKDIR. The WORKDIR must be somewhere in your home directory. If this does not work for you, follow the Singularity instructions on setting up additional bind points (version 3.7. For other versions, see the docs). Once inside the WORKDIR, clone SpECTRE into WORKDIR/SPECTRE_ROOT
.Run sudo singularity build spectre.img docker://sxscollaboration/spectre:dev
. You can also use spectre:ci instead of spectre:dev if you want more compilers installed.
If you get the error message that makesquashfs
did not have enough space to create the image you need to set a different SINGULARITY_TMPDIR
. This can be done by running: sudo SINGULARITY_TMPDIR=/path/to/new/tmp singularity build spectre.img docker://sxscollaboration/spectre:dev
. Normally SINGULARITY_TMPDIR
is /tmp
, but building the image will temporarily need almost 8GB of space.
You can control where Singularity stores the downloaded image files from DockerHub by specifying the SINGULARITY_CACHEDIR
environment variable. The default is $HOME/.singularity/
. Note that $HOME
is /root
when running using sudo
.
singularity shell spectre.img
and you will be dropped into a bash shell.Notes:
-D Python_EXECUTABLE=/path/to/python
to the cmake command where /path/to/python
is usually /usr/bin/pythonX
and X
is the version you want.singularity shell spectre.img
in another terminal.SpECTRE's dependencies can be installed with Spack, a package manager tailored for HPC use. Install Spack by cloning it into SPACK_DIR
(a directory of your choice). Then, enable Spack's shell support with source SPACK_DIR/share/spack/setup-env.sh
. Consider adding this line to your .bash_profile
, .bashrc
, or similar. Refer to Spack's getting started guide for more information.
Once you have Spack installed, one way to install the SpECTRE dependencies is with a Spack environment:
You can also install the Spack packages listed in the environment file above with a plain spack install
if you prefer.
Notes:
spack external find
feature and the --reuse
flag to spack concretize
(or spack install
). You can also install some of the dependencies with your system's package manager in advance, e.g., with apt
or brew
. If they are not picked up by spack external find
automatically, register them with Spack manually. See the Spack documentation on external packages for details.If you are not using a container, haven't installed Charm++ with Spack, or want to install Charm++ manually for other reasons, follow the installation instructions in the Charm++ repository and in their documentation. Here are a few notes:
git checkout v7.0.0
to switch to a supported, stable release of Charm++.${SPECTRE_ROOT}/support/Charm
. For example, if you have Charm++ v7.0.0 then the patch will be v7.0.0.patch
.LIBS
target to compile. This is needed so that we can support the more sophisticated load balancers in SpECTRE executables.multicore-linux-x86_64
, or multicore-darwin-x86_64
on macOS. On an HPC system the correct Charm++ target architecture depends on the machine's inter-node communication architecture. It might take some experimenting to figure out which Charm++ configuration provides the best performance.--build-shared
to the ./build
command or pass BUILD_SHARED=ON
to the CMake configuration (see the Charm++ installation instructions).Once you have set up your development environment you can compile SpECTRE. Follow these steps:
/work/spectre-build
. It can be useful to add a descriptive label to the name of the build directory since you may create more later, e.g., build-clang-Debug
. Then, cd
into the build directory./work/charm_7_0_0/multicore-linux-x86_64-gcc
for GCC builds and /work/charm_7_0_0/mpi-linux-x86_64-smp-clang
for clang builds. For Spack installations you can determine it with spack location --install-dir charmpp
. We refer to the install directory as CHARM_ROOT
below.cmake
command to configure the build, select compilers, etc. For instance, to build with clang you may run: make list
(or ninja list
if you are using the Ninja generator) or by using tab completion. Compile targets with make -jN TARGET
(or ninja -jN TARGET
), where N
is the number of cores to build on in parallel (e.g. -j4
). Note that the Ninja generator allows you to compile individual source files too.unit-tests
target and run ctest -L unit
to run unit tests. Compile test-executables
and run ctest
to run all tests, including executables. To compile test-executables
you may have to reduce the number of cores you build on in parallel to avoid running out of memory.cli
target (see Command-line interface (CLI)).all-pybindings
target (see Using SpECTRE's Python modules).For any coverage analysis you will need to have LCOV installed on the system. For documentation coverage analysis you will also need to install coverxygen and for test coverage analysis gcov.
If you have these installed (which is already done if you are using the docker container), you can look at code coverage as follows:
-D COVERAGE=ON
to cmake
make unit-test-coverage
docs/html/unit-test-coverage
.