SpECTRE
v2024.09.29
|
SpECTRE uses CMake for the build system. In this guide we'll outline how to configure SpECTRE and describe some commonly used CMake flags. We'll also go over how to add new source files, libraries, unit tests, executables, and external dependencies.
The command to configure SpECTRE will look like this
where FLAG{1..N}
are CMake flags that are detailed in the section on commonly used CMake flags.
CMake will look in your $PATH
environment variable for all of the build dependencies and will use the first one it finds that satisfies the requirements. However, if a dependency is not in your $PATH
, there are multiple versions of a dependency, or you just want to customize your configuration, you'll have to tell CMake which dependency you want to use and where it is using CMake flags. To that end, even though the commonly used CMake flags section has a more detailed list of the flags available, the following list has the most common CMake flags you'll need to customize your configuration (e.g. compilers, charm, build type, and library types):
The following are common flags used to control building SpECTRE with CMake (in alphabetical order):
-fsanitize=address
) (default is OFF
)ON
)BLAZE_USE_STRONG_INLINE
(see below)function inlined
, then forced inlining should be disabled.ON
)function inlined
, then strong inlining should be disabled.support/Python/requirements.txt
and support/Python/dev_requirements.txt
, respectively. This is an alternative to creating a Python environment and installing the packages yourself. If you run into problems with packages like h5py, numpy or scipy you can/should still install them yourself to make sure they use the correct HDF5, BLAS, etc. (default is OFF
)ON
)ON
)OFF
)ON
)Charm++
-tracemode projections
. (default is OFF
)-tracemode summary
. (default is OFF
)Debug
(the default if the flag is not specified): sets flags that trigger additional error checkingRelease
C
compiler used (defaults to whatever is determined by CMake/Modules/CMakeDetermineCCompiler.cmake
, usually cc
)C
compiler.C++
compiler used (defaults to whatever is determined by CMake/Modules/CMakeDetermineCXXCompiler.cmake
, usually c++
)C++
compiler.Fortran
compiler used (defaults to whatever is determined by CMake/Modules/CMakeDetermineFortranCompiler.cmake
)Fortran
compiler.install
target copies executables, libraries, etc. Make sure to set this variable before you install
, or a default location such as /usr/local
is used.<BUILD_DIR>/lib
and executables in <BUILD_DIR>/bin
.OFF
)ON
)OFF
). Requires BUILD_DOCS=ON
.OFF
)OFF
)ON
)libkadath.a
was installed. Also requires FFTW to be installed (see FUKA docs on dependencies).OFF
)MEMORY_ALLOCATOR=SYSTEM
to see if that resolves the issue. It could be the case that different third-party libraries accidentally end up using different allocators, which is undefined behavior and will result in complete chaos. (default is JEMALLOC
)OFF
)SPECTRE_DEBUG
macro to enable ASSERT
s and other debug checks so they can be used in Release builds. That is, you get sanity checks and compiler optimizations. You cannot disable the checks in Debug builds, so this option has no effect in Debug builds. (default is OFF
in release)-Oz
compiler flag). The default is OFF
, unless on Apple Silicon machines, where the default is ON
. This is because of a bug in macOS on Apple Silicon, which means executables larger than 2GB in size cannot run. Disabling this option on Apple Silicon can improve debugging tests (which are small enough so the bug does not affect them).OFF
)OFF
)libgfortran
and libquadmath
.low
(not usually run on CI), normal
(run at least once on CI), high
(run always on CI). (default is normal
)OFF
)SPECTRE_LTO=ON
and the compiler supports LTO.charmrun
, for example. (default is to not use one)1
).ON
)function inlined
, then forced inlining should be disabled.OFF
)OFF
)OFF
)-fsanitize=integer
) (default is OFF
)-fsanitize=undefined
) (default is OFF
)unit-tests
target as part of the test-executables
target. This is used to build only the non-unit tests in the CI build that doesn't use the PCH. (default is ON
)ON
)ON
)ON
)ld
, gold
, and lld
. (default is OFF
)ON
)sin
, cos
, and exp
. (default is OFF
)sin
, cos
, and exp
. Defines the macro SPECTRE_USE_XSIMD
, which can be check to enable manual vectorization where necessary. (default is OFF
)To see all possible build targets, once you configure SpECTRE run
This will be a long list of all libraries, test executables, simulation executables, and custom build targets. The custom targets that are available to build with make
or ninja
are:
ctest -L unit
. Available if BUILD_TESTING
is ON
(the default).ctest
. Available if BUILD_TESTING
is ON
(the default). To compile test-executables
you may have to reduce the number of cores you build on in parallel to avoid running out of memory.CMAKE_INSTALL_PREFIX
. Doesn't try to build anything else.CMakeLists.txt
files, it is conventional to indent multiline commands by two spaces (except for the first line), and to separate most commands by blank lines.SpECTRE organizes source files into subdirectories of src
that are compiled into libraries. To add a new source file FILE.cpp
to an existing library in src/PATH/DIR
, just edit src/PATH/DIR/CMakeLists.txt
and add FILE.cpp
to the list of files in
such that the resulting <list_of_files>
is in alphabetical order.
Similarly to adding new source files, you can add a new header file FILE.hpp
to a library by editing the CMakeLists.txt
in that directory and adding FILE.hpp
to the list of files in
.tpp
files also go in this list of header files.To add a source file FILE.cpp
that is compiled into a new library LIB
in a directory src/PATH/DIR
(either in a new directory, or in an existing directory that either does not have a CMakeLists.txt
file, or does not create a library in the existing CMakeLists.txt
):
CMakeLists.txt
file in DIR
, with the following two lines at the top: src/PATH
), (if necessary) add the following line to its CMakeLists.txt
file (if necessary, recursively do the previous step and this one until you reach a CMakeLists.txt
that adds the appropriate subdirectory): add_subdirectory()
lines in the file, place the new one so that the subdirectories are in alphabetical order.LIB
= DIR
. As library names must be unique, this is not always possible, in which case the convention is to prepend the parent directory to DIR
.<list_of_X_libraries>
is an alphabetized list of libraries of the form INTERFACE
are those included in at least one .hpp
file in LIB
but never used in any .cpp
files in LIB
. The libraries listed under PRIVATE
are used in at least one .cpp
file in LIB
but not in any .hpp
file in LIB
. The libraries listed under PUBLIC
are used in at least one .hpp
file and at least one .cpp
file in LIB
. Note that a library counts as being used in a .cpp
file if the corresponding .hpp
file includes it. In other words, list a dependency as PRIVATE
if it is needed only to compile the library, but not for including headers. List a dependency as INTERFACE
if it is not needed to compile the library, but is needed for including headers. List a dependency as PUBLIC
if it is needed for both.INTERFACE
like so in the add_spectre_library(${LIBRARY} INTERFACE)
function. You should also omit the spectre_target_sources
lines as well.We use the Catch testing framework for unit tests. All unit tests are housed in tests/Unit
with subdirectories for each subdirectory of src
. Add the cpp
file to the appropriate subdirectory and also to the CMakeLists.txt
in that subdirectory. Inside the source file you can create a new test by adding a SPECTRE_TEST_CASE("Unit.Dir.Component", "[Unit][Dir][Tag]")
. The [Tag]
is optional and you can have more than one, but the tags should be used quite sparingly. The purpose of the tags is to be able to run all unit tests or all tests of a particular set of components, e.g. ctest -L Data
to run all tests inside the Data
directory. Please see writing unit tests, other unit tests and the Catch documentation for more help on writing tests. Unit tests should take as short a time as possible, with a goal of less than two seconds. Please also limit the number of distinct cases (by using SECTION
s).
You can check the unit test coverage of your code by installing all the optional components and then running make unit-test-coverage
(after re-running CMake). This will create the directory BUILD_DIR/docs/html/unit-test-coverage/
which is where the coverage information is located. Open the index.html
file in your browser and make sure that your tests are indeed checking all lines of your code. Your pull requests might not be merged until your line coverage is over 90% (we are aiming for 100% line coverage wherever possible). Unreachable lines of code can be excluded from coverage analysis by adding the inline comment LCOV_EXCL_LINE
or a block can be excluded using LCOV_EXCL_START
and LCOV_EXCL_STOP
. However, this should be used extremely sparingly since unreachable code paths should be removed from the code base altogether.
All general executables are found in src/Executables
, while those for specific evolution (elliptic) systems are found in src/Evolution/Executables
(src/Elliptic/Executables
). See how to create executables.
To add an external dependency, first add a SetupDEPENDENCY.cmake
file to the cmake
directory. If CMake does not already support find_package
for the library you're adding you can write your own FindDEPENDENCY.cmake
file. The SetupDEPENDENCY.cmake
file must then be included in the root spectre/CMakeLists.txt
. Be sure to test both that setting LIBRARY_ROOT
works correctly for your library, and also that if the library is required that CMake fails gracefully if the library is not found.
Getting dependencies of libraries correct is quite difficult. SpECTRE offers the CMake function check_spectre_libs_dependencies
, defined in cmake/SpectreCheckDependencies.cmake
, to check the dependencies for all libraries in the libs
target. Individual target dependencies can be checked using the check_target_dependencies
CMake function defined in cmake/SpectreCheckTargetDependencies.cmake
. Please see those functions in the source tree for more details on how to use them.
SpECTRE's implementation of Formaline is based on, but distinct in implementation from, the original design which embeds an archive of the source tree into the executable. The original design creates a C/C++ file with a function that returns an array/vector of char
s (a byte stream). However, this results in a very large source file (50MB or more), which is very slow to compile and ends up more than doubling the link time. Instead, SpECTRE's Formaline implementation uses the linker ld
to encode a file into an object, which means rather than creating a large source file, we can directly encode the source tree archive into the binary at the linking stage.
Most of SpECTRE's Formaline is implemented inside the tools/WrapExecutableLinker.sh
script. Function declarations are provided in Utilities/Formaline.hpp
and a small function that writes the source file to disk is defined in Utilities/Formaline.cpp
. The first Formaline-related thing done in WrapExecutableLinker.sh
is to archive everything in the source directory tracked by git. Once the archive is created we run ld -r -b binary -o object.o src.tar.gz
(with unique names for object.o
and src.tar.gz
for each executable that is built to avoid name collisions) to generate an object file with the source file encoded from _binary_src_tar_gz_start
to _binary_src_tar_gz_end
. Next we write a C++ source file that defines a function get_archive
to convert the byte stream into a std::vector<char>
. We also encode the output of printenv
, the various PATH
environment variables, and the CMake generated BuildInfo.txt
file into the source file. Finally, the generated source file is built during the linking phase and the object file containing the source archive is linked into the executable.
To further aid in reproducibility, the printenv
output and BuildInfo.txt
contents are written to HDF5 files as part of the h5::Header
object. The archive of the source tree is written using the h5::SourceArchive
object and can be extracted by running
Kokkos support is still extremely experimental, but can be enabled by passing -D SPECTRE_KOKKOS=ON
to CMake. You must pass additional CMake flags that Kokkos will use to configure itself. For example, to enable CUDA support you must pass -D Kokkos_ENABLE_CUDA=ON
. See the Kokkos documentation for details.
If you are using CUDA to compile for Nvidia GPUs but do not have the target GPU on the system you are compiling on then you must also tell CMake what CUDA architecture to use. You can do this by passing -D CMAKE_CUDA_ARCHITECTURES=80
to CMake. You must choose the architecture that will be compatible with the GPU you plan to use. Note that this is actually called the compute capability version by Nvidia and can be viewed here.