Line data Source code
1 0 : \cond NEVER
2 : Distributed under the MIT License.
3 : See LICENSE.txt for details.
4 : \endcond
5 : # Build System {#spectre_build_system}
6 :
7 : \tableofcontents
8 :
9 : # CMake {#cmake}
10 :
11 : SpECTRE uses [CMake](https://cmake.org/) for the build system. In this
12 : section of the guide we outline how to [add new source
13 : files](#adding_source_files), [libraries](#adding_libraries), [unit
14 : tests](#adding_unit_tests), [executables](#adding_executables), and
15 : [external dependencies](#adding_external_dependencies). We also
16 : describe [commonly used CMake flags](#common_cmake_flags).
17 :
18 : Note that in editing `CMakeLists.txt` files, it is conventional to
19 : indent multiline commands by two spaces (except for the first line),
20 : and to separate most commands by blank lines.
21 :
22 : ## Adding Source Files {#adding_source_files}
23 :
24 : SpECTRE organizes source files into subdirectories of `src` that are
25 : compiled into libraries. To add a new source file `FILE.cpp` to an
26 : existing library in `src/PATH/DIR`, just edit
27 : `src/PATH/DIR/CMakeLists.txt` and add `FILE.cpp` to the list of files
28 : in
29 : ```
30 : set(LIBRARY_SOURCES
31 : <list_of_files>
32 : )
33 : ```
34 : such that the resulting `<list_of_files>` is in alphabetical order.
35 :
36 : ### Adding Libraries {#adding_libraries}
37 :
38 : To add a source file `FILE.cpp` that is compiled into a new library `LIB` in a
39 : directory `src/PATH/DIR` (either in a new directory, or in an existing
40 : directory that either does not have a `CMakeLists.txt` file, or does
41 : not create a library in the existing `CMakeLists.txt`):
42 : - Create (if necessary) a `CMakeLists.txt` file in `DIR`, with the following
43 : two lines at the top:
44 : ```
45 : # Distributed under the MIT License.
46 : # See LICENSE.txt for details.
47 : ```
48 : - In the parent directory (i.e. `src/PATH`), (if necessary) add the
49 : following line to its `CMakeLists.txt` file (if necessary, recursively
50 : do the previous step and this one until you reach a `CMakeLists.txt` that
51 : adds the appropriate subdirectory):
52 : ```
53 : add_subdirectory(DIR)
54 : ```
55 : If there are already other `add_subdirectory()` lines in the file, place
56 : the new one so that the subdirectories are in alphabetical order.
57 : - Add the line:
58 : ```
59 : set(LIBRARY LIB)
60 : ```
61 : where convention is that `LIB` = `DIR`. As library names must be
62 : unique, this is not always possible, in which case the convention is to
63 : prepend the parent directory to `DIR`.
64 : - Add the lines
65 : ```
66 : set(LIBRARY_SOURCES
67 : FILE.cpp
68 : )
69 :
70 : add_spectre_library(${LIBRARY} ${LIBRARY_SOURCES})
71 :
72 : target_link_libraries(
73 : ${LIBRARY}
74 : PUBLIC
75 : <list_of_public_libraries>
76 : PRIVATE
77 : <list_of_private_libraries>
78 : INTERFACE
79 : <list_of_interface_libraries>
80 : )
81 : ```
82 : where each `<list_of_X_libraries>` is an alphabetized list
83 : of libraries of the form
84 : ```
85 : SomeLibrary
86 : SomeOtherLibrary
87 : YetAnotherLibrary
88 : ```
89 : The libraries listed under `INTERFACE` are those included in at
90 : least one `.hpp` file in `LIB` but never used in any `.cpp` files
91 : in `LIB`. The libraries listed under `PRIVATE` are used in
92 : at least one `.cpp` file in `LIB` but not in any `.hpp` file
93 : in `LIB`. The libraries listed under `PUBLIC` are used in at
94 : least one `.hpp` file and at least one `.cpp` file in `LIB`.
95 : Note that a library counts as being used in a `.cpp` file if the
96 : corresponding `.hpp` file includes it. In other words, list a dependency
97 : as `PRIVATE` if it is needed only to compile the library, but not for
98 : including headers. List a dependency as `INTERFACE` if it is not needed
99 : to compile the library, but is needed for including headers. List a
100 : dependency as `PUBLIC` if it is needed for both.
101 :
102 : ## Adding Unit Tests {#adding_unit_tests}
103 :
104 : We use the [Catch](https://github.com/philsquared/Catch) testing
105 : framework for unit tests. All unit tests are housed in `tests/Unit`
106 : with subdirectories for each subdirectory of `src`. Add the `cpp` file
107 : to the appropriate subdirectory and also to the `CMakeLists.txt` in
108 : that subdirectory. Inside the source file you can create a new test by
109 : adding a `SPECTRE_TEST_CASE("Unit.Dir.Component",
110 : "[Unit][Dir][Tag]")`. The `[Tag]` is optional and you can have more
111 : than one, but the tags should be used quite sparingly. The purpose of
112 : the tags is to be able to run all unit tests or all tests of a
113 : particular set of components, e.g. `ctest -L Data` to run all tests
114 : inside the `Data` directory. Please see \ref writing_unit_tests
115 : "writing unit tests", other unit tests and the [Catch
116 : documentation](https://github.com/philsquared/Catch) for more help on
117 : writing tests. Unit tests should take as short a time as possible,
118 : with a goal of less than two seconds. Please also limit the number of
119 : distinct cases (by using `SECTION`s).
120 :
121 : You can check the unit test coverage of your code by installing all the optional
122 : components and then running `make unit-test-coverage` (after re-running CMake).
123 : This will create the
124 : directory `BUILD_DIR/docs/html/unit-test-coverage/` which is where the coverage
125 : information is located. Open the `index.html` file in your browser and make
126 : sure that your tests are indeed checking all lines of your code. Your pull
127 : requests might not be merged until your line coverage is over 90% (we are aiming
128 : for 100% line coverage wherever possible). Unreachable lines of code can be
129 : excluded from coverage analysis by adding the inline comment `LCOV_EXCL_LINE`
130 : or a block can be excluded using `LCOV_EXCL_START` and `LCOV_EXCL_STOP`.
131 : However, this should be used extremely sparingly since unreachable code paths
132 : should be removed from the code base altogether.
133 :
134 : ## Adding Executables {#adding_executables}
135 :
136 : All general executables are found in `src/Executables`, while those
137 : for specific evolution (elliptic) systems are found in
138 : `src/Evolution/Executables` (`src/Elliptic/Executables`). See \ref
139 : dev_guide_creating_executables "how to create executables".
140 :
141 : ## Adding External Dependencies {#adding_external_dependencies}
142 :
143 : To add an external dependency, first add a `SetupDEPENDENCY.cmake`
144 : file to the `cmake` directory. You should model this after the
145 : existing one for `Brigand` if you're adding a header-only
146 : library and `yaml-cpp` if the library is not header-only. If CMake
147 : does not already support `find_package` for the library you're adding
148 : you can write your own. These should be modeled after `FindBrigand`
149 : for header-only libraries, and `FindYAMLCPP` for compiled
150 : libraries. The `SetupDEPENDENCY.cmake` file must then be included in
151 : the root `spectre/CMakeLists.txt`. Be sure to test both that setting
152 : `LIBRARY_ROOT` works correctly for your library, and also that if the
153 : library is required that CMake fails gracefully if the library is not
154 : found.
155 :
156 : ## Commonly Used CMake flags {#common_cmake_flags}
157 : The following are the most common flags used to control building with
158 : `CMake`. They are used by
159 : ```
160 : cmake -D FLAG1=OPT1 ... -D FLAGN=OPTN <SPECTRE_ROOT>
161 : ```
162 : - ASAN
163 : - Whether or not to turn on the address sanitizer compile flags
164 : (`-fsanitize=address`) (default is `OFF`)
165 : - BLAZE_USE_ALWAYS_INLINE
166 : - Force Blaze inlining (default is `ON`)
167 : - If disabled or if the platform is unable to 100% guarantee inlining,
168 : falls back to `BLAZE_USE_STRONG_INLINE` (see below)
169 : - Forced inlining reduces function call overhead, and so generally reduces
170 : runtime. However, it does increase compile time and compile memory usage. It
171 : is also easier to use a debugger when forced inlining is disabled. If you
172 : are encountering debugger messages like `function inlined`, then forced
173 : inlining should be disabled.
174 : - BLAZE_USE_STRONG_INLINE
175 : - Increase the likelihood of Blaze inlining (default is `ON`)
176 : - Strong inlining reduces function call overhead, and so generally reduces
177 : runtime. However, it does increase compile time and compile memory usage. It
178 : is also easier to use a debugger when strong inlining is disabled. If you
179 : are encountering debugger messages like `function inlined`, then strong
180 : inlining should be disabled.
181 : - BOOTSTRAP_PY_DEPS and BOOTSTRAP_PY_DEV_DEPS
182 : - Install missing Python dependencies into the build directory, as listed in
183 : `support/Python/requirements.txt` and `support/Python/dev_requirements.txt`,
184 : respectively. This is an alternative to creating a Python environment and
185 : installing the packages yourself. If you run into problems with packages
186 : like h5py, numpy or scipy you can/should still install them yourself to make
187 : sure they use the correct HDF5, BLAS, etc.
188 : (default is `OFF`)
189 : - BUILD_PYTHON_BINDINGS
190 : - Build python libraries to call SpECTRE C++ code from python
191 : (default is `ON`)
192 : - BUILD_SHARED_LIBS
193 : - Whether shared libraries are built instead of static libraries
194 : (default is `OFF`)
195 : - BUILD_TESTING
196 : - Enable building tests. (default is `ON`)
197 : - BUILD_DOCS
198 : - Enable building documentation. (default is `ON`)
199 : - DOCS_ONLY
200 : - Build _only_ documentation (default is `OFF`). Requires `BUILD_DOCS=ON`.
201 : - CHARM_ROOT
202 : - The path to the build directory of `Charm++`
203 : - CHARM_TRACE_PROJECTIONS
204 : - Enables tracing with Charm++ projections. Specifically, enables the link
205 : flag `-tracemode projections`. (default is `OFF`)
206 : - CHARM_TRACE_SUMMARY
207 : - Enables trace summaries with Charm++ projections. Specifically, enables
208 : the link flag `-tracemode summary`. (default is `OFF`)
209 : - CMAKE_BUILD_TYPE
210 : - Sets the build type. Common options:
211 : - `Debug` (the default if the flag is not specified): sets flags
212 : that trigger additional error checking
213 : - `Release`
214 : - CMAKE_C_COMPILER
215 : - The `C` compiler used (defaults to whatever is determined by
216 : `CMake/Modules/CMakeDetermineCCompiler.cmake`, usually `cc`)
217 : - CMAKE_CXX_COMPILER
218 : - The `C++` compiler used (defaults to whatever is determined by
219 : `CMake/Modules/CMakeDetermineCXXCompiler.cmake`, usually `c++`)
220 : - CMAKE_Fortran_COMPILER
221 : - The `Fortran` compiler used (defaults to whatever is determined by
222 : `CMake/Modules/CMakeDetermineFortranCompiler.cmake`)
223 : - CMAKE_C_FLAGS
224 : - Additional flags passed to the `C` compiler.
225 : - CMAKE_CXX_FLAGS
226 : - Additional flags passed to the `C++` compiler.
227 : - CMAKE_Fortran_FLAGS
228 : - Additional flags passed to the `Fortran` compiler.
229 : - CMAKE_RUNTIME_OUTPUT_DIRECTORY
230 : - Sets the directory where the library and executables are placed.
231 : By default libraries end up in `<BUILD_DIR>/lib` and executables
232 : in `<BUILD_DIR>/bin`.
233 : - CMAKE_INSTALL_PREFIX
234 : - Location where the `install` target copies executables, libraries, etc. Make
235 : sure to set this variable before you `install`, or a default location such
236 : as `/usr/local` is used.
237 : - COVERAGE
238 : - Enable code coverage with GCOV and LCOV (default `OFF`)
239 : - DEBUG_SYMBOLS
240 : - Whether or not to use debug symbols (default is `ON`)
241 : - Disabling debug symbols will reduce compile time and total size of the build
242 : directory.
243 : - ENABLE_OPENMP
244 : - Enable OpenMP parallelization in some parts of the code, such as Python
245 : bindings and interpolating volume data files. Note that simulations do not
246 : typically use OpenMP parallelization, so this flag only applies to tools.
247 : - ENABLE_PARAVIEW
248 : - Try to find ParaView to enable 3D rendering tools (default is `OFF`)
249 : - ENABLE_PROFILING
250 : - Enables various options to make profiling SpECTRE easier
251 : (default is `OFF`)
252 : - ENABLE_SPECTRE_DEBUG
253 : - Defines `SPECTRE_DEBUG` macro to enable `ASSERT`s and other debug
254 : checks so they can be used in Release builds. That is, you get sanity checks
255 : and compiler optimizations. You cannot disable the checks in Debug builds,
256 : so this option has no effect in Debug builds.
257 : (default is `OFF` in release)
258 : - ENABLE_WARNINGS
259 : - Whether or not warning flags are enabled (default is `ON`)
260 : - FUKA_ROOT
261 : - Set to a path to a [FUKA](https://bitbucket.org/fukaws/fuka) installation to
262 : enable loading FUKA initial data into SpECTRE. Can be the FUKA repository
263 : root or the directory where `libkadath.a` was installed. Also requires FFTW
264 : to be installed (see FUKA docs on dependencies).
265 : - KEEP_FRAME_POINTER
266 : - Whether to keep the frame pointer. Needed for profiling or other cases
267 : where you need to be able to figure out what the call stack is.
268 : (default is `OFF`)
269 : - MACHINE
270 : - Select a machine that we know how to run on, such as a particular
271 : supercomputer. A file named MACHINE.yaml must exist in support/Machines and
272 : a submit script template named MACHINE.sh must exist in
273 : support/SubmitScripts.
274 : - MEMORY_ALLOCATOR
275 : - Set which memory allocator to use. If there are unexplained segfaults or
276 : other memory issues, it would be worth setting `MEMORY_ALLOCATOR=SYSTEM` to
277 : see if that resolves the issue. It could be the case that different
278 : third-party libraries accidentally end up using different allocators, which
279 : is undefined behavior and will result in complete chaos.
280 : (default is `JEMALLOC`)
281 : - PY_DEV_MODE
282 : - Enable development mode for the Python package, meaning that Python files
283 : are symlinked rather than copied to the build directory. Allows to edit and
284 : test Python code much easier, in particular when it uses compiled Python
285 : bindings, but doesn't replace CMake placeholders in the Python code such as
286 : the project version. (default is `OFF`)
287 : - SPEC_ROOT
288 : - Set to a path to a SpEC installation (the SpEC repository root) to link in
289 : SpEC libraries. In particular, the SpEC::Exporter library is linked in and
290 : enables loading SpEC data into SpECTRE. See \ref installation for details.
291 : - SPECTRE_INPUT_FILE_TEST_MIN_PRIORITY
292 : - Minimum priority of input file tests to run. Possible values are: `low` (not
293 : usually run on CI), `normal` (run at least once on CI), `high` (run always
294 : on CI). (default is `normal`)
295 : - SPECTRE_LTO
296 : - Enable link-time optimization if the compiler supports it.
297 : - SPECTRE_LTO_CORES
298 : - Specifies the number of cores to use for parallelizing LTO. Must be a
299 : positive integer or "auto". This is only available when `SPECTRE_LTO=ON` and
300 : the compiler supports LTO.
301 : - SPECTRE_TEST_RUNNER
302 : - Run test executables through a wrapper. This might be `charmrun`, for
303 : example. (default is to not use one)
304 : - SPECTRE_TEST_TIMEOUT_FACTOR (and specific overrides
305 : SPECTRE_X_TEST_TIMEOUT_FACTOR for X one of UNIT, STANDALONE, INPUT_FILE, or
306 : PYTHON)
307 : - Multiply the timeout for the respective set of tests by this factor (default
308 : is `1`).
309 : - This is useful to run tests on slower machines.
310 : - SPECTRE_USE_ALWAYS_INLINE
311 : - Force SpECTRE inlining (default is `ON`)
312 : - Forced inlining reduces function call overhead, and so generally reduces
313 : runtime. However, it does increase compile time and compile memory usage. It
314 : is also easier to use a debugger when forced inlining is disabled. If you
315 : are encountering debugger messages like `function inlined`, then forced
316 : inlining should be disabled.
317 : - STRIP_SYMBOLS
318 : - Whether or not to strip all symbols (default is `OFF`)
319 : - If enabled strips all extraneous symbols from libraries and executables,
320 : further reducing the size of them.
321 : - STUB_EXECUTABLE_OBJECT_FILES
322 : - Replace object files from executables after linking with empty stubs
323 : (default is `OFF`)
324 : - This is useful for drastically reducing the build size in CI, but since the
325 : object files are replaced with empty stubs will generally cause linking
326 : problems if used during development.
327 : - STUB_LIBRARY_OBJECT_FILES
328 : - Replace object files from libraries after linking with empty stubs
329 : (default is `OFF`)
330 : - This is useful for drastically reducing the build size in CI, but since the
331 : object files are replaced with empty stubs will generally cause linking
332 : problems if used during development.
333 : - UBSAN_INTEGER
334 : - Whether or not to turn on the undefined behavior sanitizer
335 : [unsigned integer
336 : overflow](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) flag
337 : (`-fsanitize=integer`) (default is `OFF`)
338 : - UBSAN_UNDEFINED
339 : - Whether or not to turn on the undefined behavior sanitizer
340 : [undefined
341 : behavior](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html)
342 : compile flags (`-fsanitize=undefined`) (default is `OFF`)
343 : - UNIT_TESTS_IN_TEST_EXECUTABLES
344 : - Whether to build the `unit-tests` target as part of the `test-executables`
345 : target. This is used to build only the non-unit tests in the CI build
346 : that doesn't use the PCH. (default is `ON`)
347 : - USE_CCACHE
348 : - Use ccache to cache build output so that rebuilding parts of the source tree
349 : is faster. The cache will use up space on disk, with the default being
350 : around 2-5GB. If you are performing a one time build to test something
351 : specific you should consider disabling ccache in order to avoid removing
352 : cached files that may be useful in other builds.
353 : (default is `ON`)
354 : - USE_FORMALINE
355 : - Write the source tree into HDF5 files written to disk in order to increase
356 : reproducibility of results.
357 : (default is `ON`)
358 : - USE_GIT_HOOKS
359 : - Use git hooks to perform some sanity checks so that small goofs are caught
360 : before they are committed. These checks are particularly useful because they
361 : also run automatically on \ref github_actions_guide "CI" and must pass
362 : before pull requests are merged.
363 : (default is `ON`)
364 : - USE_IWYU
365 : - Enable [include-what-you-use (IWYU)](https://github.com/include-what-you-use/include-what-you-use)
366 : tools. (default is `OFF`)
367 : - USE_LD
368 : - Override the automatically chosen linker. The options are `ld`, `gold`, and
369 : `lld`.
370 : (default is `OFF`)
371 : - USE_PCH
372 : - Whether or not to use pre-compiled headers (default is `ON`)
373 : - This needs to be turned `OFF` in order to use
374 : [include-what-you-use
375 : (IWYU)](https://github.com/include-what-you-use/include-what-you-use)
376 : - USE_SLEEF
377 : - Whether to use [Sleef](https://github.com/shibatch/sleef) with Blaze to
378 : vectorize addition math functions like `sin`, `cos`, and `exp`.
379 : (default is `OFF`)
380 : - \note Blaze isn't tested super thoroughly across different architectures so
381 : there's unfortunately no guarantee that Blaze+Sleef will work everywhere.
382 : - USE_XSIMD
383 : - Whether to use [xsimd](https://github.com/xtensor-stack/xsimd) with Blaze to
384 : vectorize addition math functions like `sin`, `cos`, and `exp`.
385 : Defines the macro `SPECTRE_USE_XSIMD`, which can be check to enable manual
386 : vectorization where necessary.
387 : (default is `OFF`)
388 :
389 : ## CMake targets
390 :
391 : In addition to individual simulation executables, the following targets are
392 : available to build with `make` or `ninja`:
393 :
394 : - unit-tests
395 : - Build unit tests, which you can run with `ctest -L unit`. Available if
396 : `BUILD_TESTING` is `ON` (the default).
397 : - test-executables
398 : - Build all tests, including executables, so you can run all tests with
399 : `ctest`. Available if `BUILD_TESTING` is `ON` (the default). To compile
400 : `test-executables` you may have to reduce the number of cores you build on
401 : in parallel to avoid running out of memory.
402 : - all-pybindings
403 : - Build Python bindings. See \ref spectre_using_python for details.
404 : - cli
405 : - Same as all-pybindings
406 : - install
407 : - Install targets that have been built to the `CMAKE_INSTALL_PREFIX`. Doesn't
408 : try to build anything else.
409 :
410 : ## Checking Dependencies
411 :
412 : Getting dependencies of libraries correct is quite difficult. SpECTRE offers the
413 : CMake function `check_spectre_libs_dependencies`, defined in
414 : `cmake/SpectreCheckDependencies.cmake`, to check the dependencies for all
415 : libraries in the `libs` target. Individual target dependencies can be checked
416 : using the `check_target_dependencies` CMake function defined in
417 : `cmake/SpectreCheckTargetDependencies.cmake`. Please see those functions in the
418 : source tree for more details on how to use them.
419 :
420 : ## Formaline
421 :
422 : SpECTRE's implementation of Formaline is based on, but distinct in
423 : implementation from, the original design by
424 : [Erik Schnetter and Christian Ott](https://github.com/hypercott/formaline),
425 : which embeds an archive of the source tree into the executable. The original
426 : design creates a C/C++ file with a function that returns an array/vector of
427 : `char`s (a byte stream). However, this results in a very large source file (50MB
428 : or more), which is very slow to compile and ends up more than doubling the link
429 : time. Instead, SpECTRE's Formaline implementation uses the linker `ld` to
430 : encode a file into an object, which means
431 : rather than creating a large source file, we can directly encode the source tree
432 : archive into the binary at the linking stage.
433 :
434 : Most of SpECTRE's Formaline is implemented
435 : inside the `tools/WrapExecutableLinker.sh` script. Function declarations are
436 : provided in `Utilities/Formaline.hpp` and a small function that writes the
437 : source file to disk is defined in `Utilities/Formaline.cpp`. The first
438 : Formaline-related thing done in `WrapExecutableLinker.sh` is to archive
439 : everything in the source directory tracked by git. Once the archive is created
440 : we run `ld -r -b binary -o object.o src.tar.gz` (with unique names for
441 : `object.o` and `src.tar.gz` for each executable that is built to avoid name
442 : collisions) to generate an object file with the source file encoded from
443 : `_binary_src_tar_gz_start` to `_binary_src_tar_gz_end`. Next we write a C++
444 : source file that defines a function `get_archive` to convert the byte stream
445 : into a `std::vector<char>`. We also encode the output of `printenv`, the various
446 : `PATH` environment variables, and the CMake generated `BuildInfo.txt` file
447 : into the source file. Finally, the generated source file is built during the
448 : linking phase and the object file containing the source archive is linked into
449 : the executable.
450 :
451 : To further aid in reproducibility, the `printenv` output and
452 : `BuildInfo.txt` contents are written to HDF5 files as part of the
453 : `h5::Header` object. The archive of the source tree is written using the
454 : `h5::SourceArchive` object and can be extracted by running
455 : ```
456 : h5dump -d /src.tar.gz -b LE -o src.tar.gz /path/to/hdf5/file.h5
457 : ```
|
