SpECTRE  v2024.05.11
Creating Executables

There are several different types of executables that can be built:

  • An executable that uses Charm++ to run in parallel, e.g. Executables/ParallelInfo
  • An executable that does not use Charm++, does not run in parallel, and supplies its own main, e.g. Executables/Examples/HelloWorldNoCharm
  • An executable that uses Charm++ to run in parallel but supplies its own main
  • An executable that uses custom compilation or linking flags, e.g. DebugPreprocessor
  • Executables used for evolutions or elliptic solves

Executable Using Charm++ for Parallelization

The minimal SpECTRE executable tutorial describes how to add a new parallel executable.

Another simple example of an executable using Charm++ for parallelization is in src/Executables/Examples/HelloWorld. In this example, the only additional phase (besides Initialization and Exit) is Execute, and the phases are executed in order. SingletonHelloWorld defines a single component HelloWorld

template <class Metavariables>
struct HelloWorld {
using const_global_cache_tags = tmpl::list<Tags::Name>;
using chare_type = Parallel::Algorithms::Singleton;
using metavariables = Metavariables;
using phase_dependent_action_list = tmpl::list<
using simple_tags_from_options = Parallel::get_simple_tags_from_options<
static void execute_next_phase(
const Parallel::Phase next_phase,
Parallel::CProxy_GlobalCache<Metavariables>& global_cache);
template <class Metavariables>
void HelloWorld<Metavariables>::execute_next_phase(
const Parallel::Phase /* next_phase */,
Parallel::CProxy_GlobalCache<Metavariables>& global_cache) {
The possible phases of an executable.
Definition: Phase.hpp:40
tmpl::remove_duplicates< tmpl::flatten< tmpl::transform< InitializationActionsList, detail::get_simple_tags_from_options_from_action< tmpl::_1 > > > > get_simple_tags_from_options
Given a list of initialization actions, returns a list of the unique simple_tags_from_options for all...
Definition: ParallelComponentHelpers.hpp:296
tmpl::flatten< tmpl::transform< PhaseDepActionList, detail::get_initialization_actions_list< tmpl::_1 > > > get_initialization_actions_list
Given the phase dependent action list, return the list of actions in the Initialization phase (or an ...
Definition: ParallelComponentHelpers.hpp:274
auto * local_branch(Proxy &&proxy)
Wrapper for calling Charm++'s .ckLocalBranch() on a proxy.
Definition: Local.hpp:48
A struct that stores the charm++ types relevant for a particular singleton component.
Definition: AlgorithmSingletonDeclarations.hpp:31
List of all the actions to be executed in the specified phase.
Definition: PhaseDependentActionList.hpp:17

which specifies via the chare_type type alias that it is a singleton parallel component which means that only one such object will exist across all processors used by the executable. Each component must define the static function execute_next_phase which is executed during the phases (other than Initialization and Exit) defined in the metavariables struct. In SingletonHelloWorld, the PrintMessage action is called during the Execute phase.

namespace Actions {
struct PrintMessage {
template <typename ParallelComponent, typename DbTags, typename Metavariables,
typename ArrayIndex>
static void apply(db::DataBox<DbTags>& /*box*/,
const ArrayIndex& /*array_index*/) {
Parallel::printf("Hello %s from process %d on node %d!\n",
Parallel::get<Tags::Name>(cache), sys::my_proc(),
} // namespace Actions
A Charm++ chare that caches global data once per Charm++ node.
Definition: GlobalCache.hpp:221
auto apply(F &&f, const ObservationBox< ComputeTagsList, DataBoxType > &observation_box, Args &&... args)
Apply the function object f using its nested argument_tags list of tags.
Definition: ObservationBox.hpp:238
T my_node(const DistribObject &distributed_object)
Index of my node.
Definition: Info.hpp:53
T my_proc(const DistribObject &distributed_object)
Index of my processing element.
Definition: Info.hpp:35
void printf(const std::string &format, Args &&... args)
Print an atomic message to stdout with C printf usage.
Definition: Printf.hpp:125

The PrintMessage action is executed on whatever process the singleton component is created upon, and prints a message.

Executables can read in an input file (specified by the --input-file argument) that will be parsed when the executable begins. Options specified in the input file can be used to either place items in the Parallel::GlobalCache (by specifying tags in the const_global_cache_tags type alias of the metavariables, component and action structs), to construct items in the db::DataBox of components during initialization (by specifying tags in the simple_tags_from_options type alias of action struct), or be passed to the allocate_array function of an array component (by specifying tags in the array_allocation_tags type alias of the component). SingletonHelloWorld specifies a single option

namespace OptionTags {
struct Name {
using type = std::string;
static constexpr Options::String help{"A name"};
} // namespace OptionTags
namespace Tags {
struct Name : db::SimpleTag {
using type = std::string;
using option_tags = tmpl::list<OptionTags::Name>;
static constexpr bool pass_metavariables = false;
return name;
} // namespace Tags
tuples::TaggedTuple< Tags... > create_from_options(const tuples::TaggedTuple< OptionTags... > &options, tmpl::list< Tags... >)
Given a list of tags and a tagged tuple containing items created from input options,...
Definition: CreateFromOptions.hpp:38
std::string name()
Return the result of the name() member of a class. If a class doesn't have a name() member,...
Definition: PrettyType.hpp:733
const char *const String
The string used in option structs.
Definition: String.hpp:8
Mark a struct as a simple tag by inheriting from this.
Definition: Tag.hpp:36

which a string specifying a name that will be placed into the constant global cache. The string is fetched when performing the PrintMessage action. Items in the constant global cache are stored once per node that the executable runs on. An example input file for SingletonHelloWorld can be found in tests/InputFiles/ExampleExecutables/SingletonHelloWorld.yaml and shows how to specify the options (lines beginning with a # are comments and can be ignored).

Furthermore among the included header files

#include <string>
#include "DataStructures/DataBox/DataBox.hpp"
#include "DataStructures/DataBox/Tag.hpp"
#include "Options/String.hpp"
#include "Parallel/Algorithms/AlgorithmSingleton.hpp"
#include "Parallel/CharmMain.tpp"
#include "Parallel/GlobalCache.hpp"
#include "Parallel/Invoke.hpp"
#include "Parallel/Local.hpp"
#include "Parallel/ParallelComponentHelpers.hpp"
#include "Parallel/Phase.hpp"
#include "Parallel/PhaseDependentActionList.hpp"
#include "Parallel/Printf/Printf.hpp"
#include "Utilities/System/ParallelInfo.hpp"
#include "Utilities/TMPL.hpp"
namespace PUP {
class er;
} // namespace PUP

must be the appropriate header for each parallel component type, which in the SingletonHelloWorld example is AlgorithmSingleton.hpp. Note that these headers are not in the source tree, but are generated automatically when the code is compiled.

See the Parallelization documentation for more details.

Executable Using Charm++ with Custom main()

While this is technically possible, it has not been tested. We recommend using the Charm++ supplied main chare mechanism for the time being.

Executable Not Using Charm++

An example of an executable that does not use Charm++ for parallelization but still can use all other infrastructure in SpECTRE is in src/Executables/Examples/HelloWorldNoCharm. Adding a non-Charm++ executable to SpECTRE mostly follows the standard way of adding an executable using CMake. The only deviation is that the CMakeLists.txt file must tell Charm++ not to add a main() by passing the link flags -nomain-module -nomain. This is done using CMake's set_target_properties:

PROPERTIES LINK_FLAGS "-nomain-module -nomain"

To add the executable as a target you must use the add_spectre_executable function, which is a light weight wrapper around CMake's add_executable. For example,

EXCLUDE_FROM_ALL # Exclude from calls to `make` without a specified target

You can link in any of the SpECTRE libraries by adding them to the target_link_libraries, for example:


We recommend that you add a test that the executable properly runs by adding an input file to tests/InputFiles in an appropriate subdirectory. See [tests/InputFiles/ExampleExecutables/HelloWorldNoCharm.yaml] (https://github.com/sxs-collaboration/spectre/tree/develop/tests/InputFiles/ ExampleExecutables/HelloWorldNoCharm.yaml) for an example. The input file is passed to the executable using --input-file path/to/Input.yaml. In the case of the executable not taking any input file this is just used to generate a test that runs the executable.

For these types of executables main can take the usual (int argc, char *argv[]) and parse command line options. Executables not using Charm++ are just standard executables that can link in any of the libraries in SpECTRE.

Currently calling Parallel::abort results in a segfault deep inside Charm++ code. However, the error messages from ASSERT and ERROR are still printed.

Executable With Custom Compilation or Linking Flags

Use the CMake function set_target_properties to add flags to an executable. To call a completely custom compiler invocation you should use the add_custom_target CMake function. The need for the custom_target level of control is rare and should generally be avoided since it adds quite a bit of technical debt to the code base. Thus, it is not explained here. If you are certain you need it you can see the DebugPreprocessor executable's CMakeLists.txt file for an example.

Executable Used for Evolution or Elliptic Solve

Once they are written, see the tutorials specific to evolution and elliptic solves.