SpECTRE Documentation Coverage Report
Current view: top level - __w/spectre/spectre/docs/DevGuide - ProfilingWithProjections.md Hit Total Coverage
Commit: f1ddee3e40d81480e49140855d2b0e66fafaa908 Lines: 0 1 0.0 %
Date: 2020-12-02 17:35:08
Legend: Lines: hit not hit

          Line data    Source code
       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # Profiling With Charm++ Projections {#profiling_with_projections}
       6             : 
       7             : ## Basic Setup and Compilation
       8             : 
       9             : To view trace data after a profiling run you must download Charm++'s
      10             : Projections software from their [website](http://charm.cs.illinois.edu/).
      11             : If you encounter issues it may
      12             : be necessary to clone the git repository and build the correct version
      13             : from scratch. Note that the version of Charm++ used to compile SpECTRE
      14             : should match the version of Projections used to analyze the trace data.
      15             : You can collect the trace data on a different machine than the one you
      16             : will be analyzing the data on. For example, you can collect the data on
      17             : a supercomputer and analyze it on your desktop or laptop.
      18             : 
      19             : For profiling you will want to use a production build of Charm++, which
      20             : means compiling Charm++ with the `--with-production` flag. To enable trace
      21             : collecting you must build with the `--enable-tracing` flag as well. For
      22             : example, on a multicore 64-bit Linux machine the build command would be
      23             : ``` shell
      24             : ./build charm++ multicore-linux64 gcc -j8 --with-production --enable-tracing
      25             : ```
      26             : 
      27             : When building SpECTRE for profiling there are several options to control
      28             : what is profiled. First, you'll want to compile in Release mode by specifying
      29             : `-D CMAKE_BUILD_TYPE=Release`. To enable tracing with Projections you must
      30             : specify the CMake variable `-D PROJECTIONS=ON`. By default only Charm++ entry
      31             : methods will be profiled, which will not provide very much insight because
      32             : the AlgorithmChare infrastructure has only one entry method, which happens to
      33             : be a function template. You must specify a typelist of which Actions you would
      34             : like to profile before the first time `Evolution/EvolveSystem.hpp` is included
      35             : in the main executable. This typically means right after the
      36             : `the_ordered_actions_list` list is defined in the main executable. For example,
      37             : 
      38             : ```cpp
      39             : using the_ordered_actions_list = tmpl::list<
      40             :     Algorithms::CheckTriggers<the_system, the_volume_dim>,
      41             :     Algorithms::SendDataForFluxes<the_system, the_volume_dim>,
      42             :     Algorithms::ComputeVolumeDtU<the_system, the_volume_dim>,
      43             :     Algorithms::ComputeBoundaryFlux<the_system, the_volume_dim>,
      44             :     Algorithms::ImposeExternalBoundaryConditions<the_system, the_volume_dim>,
      45             :     Algorithms::ComputeU<the_system, the_volume_dim>,
      46             :     Algorithms::AdvanceSlab<the_system, the_volume_dim>,
      47             :     Algorithms::UpdateInstance<the_system, the_volume_dim>>;
      48             : 
      49             : using the_trace_actions_list = tmpl::list<
      50             :     Algorithms::CheckTriggers<the_system, the_volume_dim>,
      51             :     Algorithms::ComputeVolumeDtU<the_system, the_volume_dim>,
      52             :     Algorithms::ComputeBoundaryFlux<the_system, the_volume_dim>,
      53             :     Algorithms::ImposeExternalBoundaryConditions<the_system, the_volume_dim>,
      54             :     Algorithms::ComputeU<the_system, the_volume_dim>>;
      55             : ```
      56             : 
      57             : Actions not in `the_trace_actions_list` will not be traced but will still be
      58             : executed. The code also records time in the AlgorithmChare's `receive_data`
      59             : method that is
      60             : not taken up by Actions (typically checking if all data needed for the next
      61             : Action has been received) and also time spent saving the received data into
      62             : local memory.
      63             : 
      64             : ## Using PAPI With Projections
      65             : 
      66             : \warning Using PAPI with Projections's stat counters requires Charm++ v6.8.0
      67             : or newer.
      68             : 
      69             : It is possible to collect information from hardware counters using PAPI
      70             : for the Actions by specifying
      71             : `-D PROJECTIONS_PAPI_COUNTERS="PAPI_L1_DCM,PAPI_L2_DCM"`. That is, a comma
      72             : separated list of PAPI counters to record. To see the list of available counters
      73             : on your hardware run `papi_avail -a`. The recorded user statistics can then be
      74             : analyzed inside Charm++ Projections. It is also possible to record custom user
      75             : statistics using Charm++ by specifying `-D PROJECTIONS_USER_STATS=ON` and
      76             : defining the variable `user_stat_names` as
      77             : 
      78             : ```cpp
      79             : static constexpr std::array<const char*, 2> user_stat_names{
      80             :     {"name_1", "name_2"}};
      81             : ```
      82             : 
      83             : The variable needs to be defined before `Evolution/EvolveSystem.hpp` is
      84             : included. To record statistics of PAPI counters it is recommended you disable
      85             : counters for Actions (you can still time profile them, though) and specify the
      86             : CMake variable `-D USE_PAPI=ON`. An example of how to record statistics from
      87             : PAPI counters is given below.
      88             : 
      89             : To provide a concrete example of tracing and analyzing hardware PMUs using PAPI
      90             : for only a subset of functions in an Action we will profile the function
      91             : `ScalarWaveEquations<Dim>::%compute_volume_dt_u`. There are several helper
      92             : functions provided in `Utilities/PAPI.hpp` that will come in useful. To start
      93             : the PAPI counters and record the time at the beginning of the function we call
      94             : 
      95             : ```cpp
      96             : start_papi_counters(std::array<int, 2>{{PAPI_L1_DCM, PAPI_L2_DCM}});
      97             : const double start_time = get_time_from_papi();
      98             : ```
      99             : 
     100             : and wherever we want to finish recording we run
     101             : 
     102             : ```cpp
     103             : const double stop_time = get_time_from_papi();
     104             : auto counters = stop_papi_counters<2>();
     105             : ```
     106             : 
     107             : The template parameter to `stop_papi_counters` must be the number of PAPI
     108             : counters being recorded, two in our example. The function `get_time_from_papi`
     109             : returns the time in seconds and `stop_time - start_time` gives the elapsed time
     110             : in seconds between the two calls. Using `get_time_from_papi` it is possible to
     111             : compute FLOP/s, or any other metric related to time. Finally, to store the
     112             : counter read outs into Projections's stat counter we use
     113             : 
     114             : ```cpp
     115             : updateStat(projections_user_stat_offset, counters[0]);
     116             : updateStat(projections_user_stat_offset + 1, counters[1]);
     117             : ```
     118             : 
     119             : The variable `projections_user_stat_offset` is used to ensure that the stat
     120             : numbers used by Charm++ internally do not collide with any used in the Actions,
     121             : unless you have over 9000 Actions.
     122             : 
     123             : ## Running SpECTRE With Trace Output
     124             : 
     125             : When running SpECTRE you must specify a directory to output trace data into.
     126             : This is done by adding the command line argument `+traceroot DIR` where `DIR` is
     127             : the directory to dump the trace data into. Note that `DIR` must already exist,
     128             : the application will not create it.
     129             : For example, to run the 3D scalar wave executable on a multicore build with
     130             : tracing enabled use
     131             : 
     132             : ```shell
     133             : ./Evolve3DScalarWave +p4 +traceroot ./traces
     134             : ```
     135             : 
     136             : For more information on runtime options to
     137             : control trace data see the
     138             : [Charm++ Projections manual](http://charm.cs.illinois.edu/manuals/html/projections/1.html).
     139             : 
     140             : ## Visualizing Trace %Data In Projections
     141             : 
     142             : See the [Charm++ Projections manual](http://charm.cs.illinois.edu/manuals/html/projections/2.html)
     143             : for details.

Generated by: LCOV version 1.14