Creating Executables

There are several different types of executables that can be built:

  • An executable that uses Charm++ to run in parallel, e.g. Executables/ParallelInfo
  • An executable that does not use Charm++, does not run in parallel, and supplies its own main, e.g. Executables/Benchmark
  • An executable that uses Charm++ to run in parallel but supplies its own main
  • An executable that uses custom compilation or linking flags, e.g. DebugPreprocessor
  • Executables used for evolutions or elliptic solves

Executable Using Charm++ for Parallelization

The minimal SpECTRE executable tutorial describes how to add a new parallel executable.

Another simple example of an executable using Charm++ for parallelization is in src/Executables/Examples/HelloWorld. In this example, the only additional phase (besides Initialization and Exit) is Execute, and the phases are executed in order. SingletonHelloWorld defines a single component HelloWorld

template <class Metavariables>
struct HelloWorld {
using const_global_cache_tags = tmpl::list<Tags::Name>;
using chare_type = Parallel::Algorithms::Singleton;
using metavariables = Metavariables;
using phase_dependent_action_list = tmpl::list<
Parallel::PhaseActions<typename Metavariables::Phase,
Metavariables::Phase::Execute, tmpl::list<>>>;
using initialization_tags = Parallel::get_initialization_tags<
static void execute_next_phase(
const typename Metavariables::Phase next_phase,
Parallel::CProxy_GlobalCache<Metavariables>& global_cache) noexcept;
};
template <class Metavariables>
void HelloWorld<Metavariables>::execute_next_phase(
const typename Metavariables::Phase /* next_phase */,
Parallel::CProxy_GlobalCache<Metavariables>& global_cache) noexcept {
Parallel::simple_action<Actions::PrintMessage>(
Parallel::get_parallel_component<HelloWorld>(
*(global_cache.ckLocalBranch())));
}

which specifies via the chare_type type alias that it is a singleton parallel component which means that only one such object will exist across all processors used by the executable. Each component must define the static function execute_next_phase which is executed during the phases (other than Initialization and Exit) defined in the metavariables struct. In SingletonHelloWorld, the PrintMessage action is called during the Execute phase.

namespace Actions {
struct PrintMessage {
template <typename ParallelComponent, typename DbTags, typename Metavariables,
typename ArrayIndex>
static void apply(db::DataBox<DbTags>& /*box*/,
const ArrayIndex& /*array_index*/) {
Parallel::printf("Hello %s from process %d on node %d!\n",
Parallel::get<Tags::Name>(cache), sys::my_proc(),
}
};
} // namespace Actions

The PrintMessage action is executed on whatever process the singleton component is created upon, and prints a message.

Executables can read in an input file (specified by the --input-file argument) that will be parsed when the executable begins. Options specified in the input file can be used to either place items in the Parallel::GlobalCache (by specifying tags in the const_global_cache_tags type alias of the metavariables, component and action structs), to construct items in the db::DataBox of components during initialization (by specifying tags in the initialization_tags type alias of action struct), or be passed to the allocate_array function of an array component (by specifying tags in the allocation_tags type alias of the component). SingletonHelloWorld specifies a single option

namespace OptionTags {
struct Name {
using type = std::string;
static constexpr Options::String help{"A name"};
};
} // namespace OptionTags
namespace Tags {
struct Name : db::SimpleTag {
using type = std::string;
using option_tags = tmpl::list<OptionTags::Name>;
static constexpr bool pass_metavariables = false;
static std::string create_from_options(const std::string& name) noexcept {
return name;
}
};
} // namespace Tags

which a string specifying a name that will be placed into the constant global cache. The string is fetched when performing the PrintMessage action. Items in the constant global cache are stored once per node that the executable runs on. An example input file for SingletonHelloWorld can be found in tests/InputFiles/ExampleExecutables/SingletonHelloWorld.yaml and shows how to specify the options (lines beginning with a # are comments and can be ignored).

Furthermore among the included header files

#include "DataStructures/DataBox/Tag.hpp"
#include "Parallel/InitializationFunctions.hpp"
#include "Parallel/Invoke.hpp"
#include "Parallel/ParallelComponentHelpers.hpp"
#include "Parallel/PhaseDependentActionList.hpp"
namespace PUP {
class er;
} // namespace PUP

must be the appropriate header for each parallel component type, which in the SingletonHelloWorld example is AlgorithmSingleton.hpp. Note that these headers are not in the source tree, but are generated automatically when the code is compiled.

See the Parallelization documentation for more details.

Executable Using Charm++ with Custom main()

While this is technically possible, it has not been tested. We recommend using the Charm++ supplied main chare mechanism for the time being.

Executable Not Using Charm++

An example of an executable that does not use Charm++ for parallelization but still can use all other infrastructure in SpECTRE is in src/Executables/Examples/HelloWorldNoCharm. Adding a non-Charm++ executable to SpECTRE mostly follows the standard way of adding an executable using CMake. The only deviation is that the CMakeLists.txt file must tell Charm++ not to add a main() by passing the link flags -nomain-module -nomain. This is done using CMake's set_target_properties:

set_target_properties(
${EXECUTABLE}
PROPERTIES LINK_FLAGS "-nomain-module -nomain"
)

To add the executable as a target you must use the add_spectre_executable function, which is a light weight wrapper around CMake's add_executable. For example,

add_spectre_executable(
${EXECUTABLE}
EXCLUDE_FROM_ALL # Exclude from calls to `make` without a specified target
HelloWorld.cpp
)

You can link in any of the SpECTRE libraries by adding them to the target_link_libraries, for example:

target_link_libraries(
${EXECUTABLE}
DataStructures
)

We recommend that you add a test that the executable properly runs by adding an input file to tests/InputFiles in an appropriate subdirectory. See tests/InputFiles/ExampleExecutables/HelloWorldNoCharm.yaml for an example. The input file is passed to the executable using --input-file path/to/Input.yaml. In the case of the executable not taking any input file this is just used to generate a test that runs the executable.

For these types of executables main can take the usual (int argc, char *argv[]) and parse command line options. Executables not using Charm++ are just standard executables that can link in any of the libraries in SpECTRE.

Warning
Currently calling Parallel::abort results in a segfault deep inside Charm++ code. However, the error messages from ASSERT and ERROR are still printed.

Executable With Custom Compilation or Linking Flags

Use the CMake function set_target_properties to add flags to an executable. To call a completely custom compiler invocation you should use the add_custom_target CMake function. The need for the custom_target level of control is rare and should generally be avoided since it adds quite a bit of technical debt to the code base. Thus, it is not explained here. If you are certain you need it you can see the DebugPreprocessor executable's CMakeLists.txt file for an example.

Executable Used for Evolution or Elliptic Solve

Once they are written, see the tutorials specific to evolution and elliptic solves.

FloatingPointExceptions.hpp
std::apply
T apply(T... args)
std::string
Parallel::GlobalCache
Definition: ElementReceiveInterpPoints.hpp:15
GlobalCache.hpp
Options.hpp
Parallel::printf
void printf(const std::string &format, Args &&... args)
Print an atomic message to stdout with C printf usage.
Definition: Printf.hpp:103
ParallelInfo.hpp
db::SimpleTag
Mark a struct as a simple tag by inheriting from this.
Definition: Tag.hpp:36
Parallel::get_initialization_tags
tmpl::remove_duplicates< tmpl::flatten< tmpl::list< AllocationTagsList, tmpl::transform< InitializationActionsList, detail::get_initialization_tags_from_action< tmpl::_1 > >> >> get_initialization_tags
Given a list of initialization actions, and possibly a list of tags needed for allocation of an array...
Definition: ParallelComponentHelpers.hpp:254
Parallel::get_initialization_actions_list
tmpl::flatten< tmpl::transform< PhaseDepActionList, detail::get_initialization_actions_list< tmpl::_1 > >> get_initialization_actions_list
Given the phase dependent action list, return the list of actions in the Initialization phase (or an ...
Definition: ParallelComponentHelpers.hpp:218
Parallel::PhaseActions
List of all the actions to be executed in the specified phase.
Definition: PhaseDependentActionList.hpp:16
Parallel::create_from_options
tuples::TaggedTuple< Tags... > create_from_options(const tuples::TaggedTuple< OptionTags... > &options, tmpl::list< Tags... >) noexcept
Given a list of tags and a tagged tuple containing items created from input options,...
Definition: CreateFromOptions.hpp:40
Parallel::my_node
int my_node(const DistribObject &distributed_object) noexcept
Index of my node.
Definition: Info.hpp:51
Printf.hpp
DataBox.hpp
Parallel::my_proc
int my_proc(const DistribObject &distributed_object) noexcept
Index of my processing element.
Definition: Info.hpp:33
ActionTesting::cache
Parallel::GlobalCache< Metavariables > & cache(MockRuntimeSystem< Metavariables > &runner, const ArrayIndex &array_index) noexcept
Returns the GlobalCache of Component with index array_index.
Definition: MockRuntimeSystemFreeFunctions.hpp:382
Parallel::Algorithms::Singleton
A struct that stores the charm++ types relevant for a particular singleton component.
Definition: AlgorithmSingletonDeclarations.hpp:29
Options::String
const char *const String
The string used in option structs.
Definition: Options.hpp:32
AlgorithmSingleton.hpp
TMPL.hpp