Line data    Source code

       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # Profiling {#profiling}
       6             : 
       7             : \tableofcontents
       8             : 
       9             : There are a number of tools available for profiling, each with their own
      10             : strengths and weaknesses. This makes it difficult to recommend one "right" way
      11             : of analyzing performance using profilers. Instead, one should use a combination
      12             : of the tools to discover and eliminate performance bottle necks. Common
      13             : profilers are Charm++ Projections (tracing-based), HPCToolkit (sampling-based,
      14             : very versatile), Linux perf (sampling-based, command line only), Intel VTune
      15             : (sampling-based, works well on Intel hardware), AMD uProf (similar to Intel
      16             : VTune), and Score-P (sampling and tracing, with MPI and pthread interfacing).
      17             : 
      18             : ## Profiling with HPCToolkit {#profiling_with_hpctoolkit}
      19             : 
      20             : Follow the HPCToolkit installation instructions at
      21             : [hpctoolkit.org](http://hpctoolkit.org). The Spack
      22             : installation seems to work well. Once installed, compile your executable in
      23             : Release mode with `-D ENABLE_PROFILING=ON -D DEBUG_SYMBOLS=ON` since otherwise
      24             : you won't be able to get call stacks and source analysis. Using `-D
      25             : BUILD_SHARED_LIBS=ON` is recommended since it makes HPCToolkit a lot easier to
      26             : use. You must also use the system allocator, `-D MEMORY_ALLOCATOR=SYSTEM`. We
      27             : will work from the build directory and perform all runs and performance analysis
      28             : there.
      29             : 
      30             : First run HPCToolkit as:
      31             : ```
      32             : hpcrun -t --event CYCLES@f200 ./bin/EXEC --input-file ./Input.yaml +p1
      33             : ```
      34             : We will profile on one core, but you can profile on multiple cores as well as
      35             : multiple nodes if using MPI as the Charm++ backend. This will generate a
      36             : `hpctoolkit-EXEC-measurements` directory. Run
      37             : ```
      38             : hpcstruct -jN ./hpctoolkit-EXEC-measurements
      39             : ```
      40             : where `N` is the number of cores to run on. This will generate a mapping to line
      41             : numbers, etc. in the measurements directory.
      42             : 
      43             : \warning Skipping the `hpcstruct` step will make `hprprof` below run extremely
      44             : slowly.
      45             : 
      46             : Once the run is complete, run
      47             : ```
      48             : hpcprof hpctoolkit-EXEC-measurements
      49             : ```
      50             : or if HPCToolkit version < 2024, run
      51             : ```
      52             : hpcprof -I /path/to/spectre/src/+ hpctoolkit-EXEC-measurements
      53             : ```
      54             : Note that the `+` is a literal `+` symbol.
      55             : 
      56             : This will create the directory
      57             : ```
      58             : hpctoolkit-EXEC-database
      59             : ```
      60             : which you can view using
      61             : ```
      62             : hpcviewer ./hpctoolkit-EXEC-database
      63             : ```
      64             : 
      65             : HPCViewer will generally start you in the `Top-down view` (callgraph of
      66             : callers). You can select  `Bottom-up view` (callgraph of callees) to get a
      67             : different perspective. Whether you want to look at the callgraph of callers or
      68             : callees depends a bit on the executable, what you're looking to measure, and how
      69             : you like to think about things. The callees graph can give you a nice overview
      70             : of what the low-level things taking up a lot of time are, but certainly makes
      71             : the call stack not look like you would expect. On the right of the callgraphs
      72             : you will see `CYCLES:Sum (I)` and `CYCLES:Sum (E)`. `I` means time spent
      73             : _including_ callees, while `E` means time spent in the function itself
      74             : (exclusive time). Sorting by exclusive gives a good idea of what the hot
      75             : functions are. Here is a screenshot from HPCViewer:
      76             : 
      77             : \image html HpcViewerCallees.png "HPCViewer callgraph of callees"
      78             : 
      79             : You can see that 49.1% of inclusive time is spent in primitive recovery, and the
      80             : line after the 49.1% function is a function inside the Kastaun
      81             : recovery scheme. The `__nss_database_lookup` is some system call,
      82             : e.g. `__memcpy_avx_unaligned_erms` or `__memset_avx2_unaligned_erms`. Looking at
      83             : the calling code, e.g. `prepare_neighbor_data` gives a good hint as to what's
      84             : going on. In most cases these are memory copies or memory sets (`std::vector`
      85             : default initializes its memory, which is bad for performance). The way to fix
      86             : these bottlenecks is to avoid memory copies and `std::vector<double>` as
      87             : buffers.
      88             : 
      89             : HPCToolkit allows you to sample on a variety of different event counters instead
      90             : of just cycles. Please see the HPCToolkit manual for details.
      91             : 
      92             : ## Profiling with AMD uProf {#profiling_with_amd_uprof}
      93             : 
      94             : [AMD uProf](https://developer.amd.com/amd-uprof/) is AMD's sampling-based
      95             : profiler that makes it relatively easy to do
      96             : quite a bit of detailed performance analysis. The uProf manual is quite good and
      97             : extensive, so for the most part the reader is referred to that. However, we will
      98             : go over some basics for profiling executables and understanding the
      99             : results. Make sure to compile your executable in Release mode with
     100             : `-DENABLE_PROFILING=ON -D DEBUG_SYMBOLS=ON` since otherwise you won't be able to
     101             : get call stacks and source analysis.
     102             : 
     103             : When you open uProf you may be asked to change the kernel event paranoid
     104             : level. Once you have uProf open, select `PROFILE` at the top. Specify the
     105             : application path, options, etc. We will again run on a single core to analyze
     106             : performance. It's recommended that you set the Core Affinity in AMD uProf so
     107             : that your application isn't migrated between cores during a profiling run. Then
     108             : choose `Next` in the lower right corner. Make sure the `CPU Profile Type` is set
     109             : to `CPU Profile` at the top. We will first do a `Time-based Sampling` run (on
     110             : the left). This means uProf will interrupt the application every `N`
     111             : milliseconds and see where the application is. You typically want a few thousand
     112             : total samples to get something that's reasonably representative of your
     113             : application. Under the `Advanced %Options` make sure `Enable CSS` (on the right)
     114             : is enabled (green) and that `Enable FPO` is also enabled. Now click `Start
     115             : Profile` in the bottom right. Once the profile is complete you will be presented
     116             : with a summary outlining where your code is spending most of its time. Click
     117             : `ANALYZE` at the top to get a more detailed view. On the left you can select
     118             : between a callgraph of callees (Function HotSpots), a callgraph of callers (Call
     119             : Graph), and a few other views. Below is an example of a result from the same run
     120             : we used with HPCToolkit above.
     121             : 
     122             : \image html AmdUprofCallgraph.png "AMD uProf callgraph of callees"
     123             : 
     124             : Again we see that most of our time is spent in primitive recovery but also that
     125             : a lot of time is spent copying memory. This was grouped into
     126             : `__nss_database_lookup` in HPCToolkit. Unfortunately, getting a call stack out
     127             : of the `memcpy` doesn't always work and so while you know you're spending a lot
     128             : of time copying memory, it's not so obvious where those copies are occurring.
     129             : 
     130             : ## Profiling With Charm++ Projections {#profiling_with_projections}
     131             : 
     132             : To view trace data after a profiling run you must download Charm++'s
     133             : Projections software from their [website](http://charm.cs.illinois.edu/).
     134             : If you encounter issues it may
     135             : be necessary to clone the git repository and build the correct version
     136             : from scratch. Note that the version of Charm++ used to compile SpECTRE
     137             : should match the version of Projections used to analyze the trace data.
     138             : You can collect the trace data on a different machine than the one you
     139             : will be analyzing the data on. For example, you can collect the data on
     140             : a supercomputer and analyze it on your desktop or laptop.
     141             : 
     142             : For profiling you will want to use a production build of Charm++, which
     143             : means compiling Charm++ with the `--with-production` flag. To enable trace
     144             : collecting you must build with the `--enable-tracing` flag as well. For
     145             : example, on a multicore 64-bit Linux machine the build command would be
     146             : ``` shell
     147             : ./build LIBS multicore-linux-x86_64 gcc -j8 --with-production --enable-tracing
     148             : ```
     149             : You must build your executable in Release mode as well, specifying
     150             : `-DCMAKE_BUILD_TYPE=Release` to CMake, as well as
     151             : ```
     152             : -DCHARM_TRACE_PROJECTIONS=ON -DCHARM_TRACE_SUMMARY=ON -DENABLE_PROFILING=ON
     153             : ```
     154             : to enable SpECTRE to use Charm++'s tracing features.
     155             : 
     156             : ### Running SpECTRE With Trace Output
     157             : 
     158             : When running SpECTRE you must specify a directory to output trace data into.
     159             : This is done by adding the command line argument `+traceroot DIR` where `DIR` is
     160             : the directory to dump the trace data into. Note that `DIR` must already exist,
     161             : the application will not create it.
     162             : For example,
     163             : 
     164             : ```shell
     165             : ./bin/EXEC --input-file ./Input.yaml +p4 +traceroot ./ExecTraces
     166             : ```
     167             : You might get a warning that Charm++ had to flush the log some number of times
     168             : during the run. Flushing the log adds overhead to the execution and so affects
     169             : timing measurements. While Charm++ has the ability to manually flush the log
     170             : periodically (and therefore exclude the time it takes to flush the log from the
     171             : trace), we have not yet implemented support for this. For short executable runs
     172             : you can increase the log size by specifying `+logsize M` when running the
     173             : executable. The default log size is 1,000,000 (1000000). Note that if you
     174             : increase the log size too much you will run out of memory/RAM.
     175             : 
     176             : For more information on runtime options to
     177             : control trace data see the
     178             : [Charm++ Projections manual](http://charm.cs.illinois.edu/manuals/html/projections/1.html).
     179             : 
     180             : ### Visualizing Trace %Data In Projections
     181             : 
     182             : By default Charm++ records entry method names by using the `PRETTY_FUNCTION`
     183             : macro. This means entry method names include all class (parallel component) and
     184             : action template parameter names, including any template parameters of the
     185             : template parameters. This very quickly leads to incomprehensibly long names that
     186             : are very difficult to read in the Projections interface. We include a basic
     187             : Python executable to handle the majority of renames, but the executable supports
     188             : additional basic (textual find-replace) and regular expression
     189             : replacements via a JSON file. These additional replacements are useful for
     190             : making executable-specific renames. The Python executable is
     191             : `tools/CharmSimplifyTraces.py` and an example replacements file is
     192             : `tools/CharmTraceReplacements.json`.
     193             : 
     194             : See the [Charm++ Projections manual](http://charm.cs.illinois.edu/manuals/html/projections/2.html)
     195             : for details.
     196             : 
     197             : ## Profiling with Score-P {#profiling_with_scorep}
     198             : 
     199             : See the
     200             : [Score-P](https://www.vi-hps.org/projects/score-p/overview/overview.html)
     201             : website for installation instructions specific to Score-P.
     202             : 
     203             : To compile SpECTRE with Score-P you must invoke CMake with the environment
     204             : variable `SCOREP_WRAPPER=off` (do NOT export this variable!) and the `scorep-`
     205             : compiler wrappers. For example,
     206             : ```sh
     207             : SCOREP_WRAPPER=off cmake -DCMAKE_C_COMPILER=scorep-gcc \
     208             :     -DCMAKE_CXX_COMPILER=scorep-g++ -DCMAKE_Fortran_COMPILER=scorep-gfortran \
     209             :     -D CMAKE_BUILD_TYPE=Release -D ENABLE_PROFILING=ON \
     210             :     -D BUILD_DOCS=OFF -D BUILD_PYTHON_BINDINGS=OFF $SPECTRE_HOME
     211             : ```
     212             : where `$SPECTRE_HOME` points to your spectre source tree. You can then compile
     213             : the executable you want to profile. When running the executable you will need to
     214             : make sure Score-P has enough memory to be able to function properly. For the
     215             : EvolveGhSingleBlackHole, Score-P needs 1.5GB of memory. To enable profiling,
     216             : disable tracing, and use 1.5GB of memory for Score-P, set the environment
     217             : variables
     218             : ```sh
     219             : SCOREP_ENABLE_PROFILING=1 \
     220             :     SCOREP_ENABLE_TRACING=0 \
     221             :     SCOREP_TOTAL_MEMORY=1536000000
     222             : ```
     223             : We also need to use a filter file to keep the data at all manageable and to
     224             : avoid some currently not understood interaction between Score-P and our option
     225             : parser. The profile is in `support/Profiling/ScoreP` and named `spectre.flt`.
     226             : The final command should look something like:
     227             : ```sh
     228             : SCOREP_ENABLE_PROFILING=1 SCOREP_ENABLE_TRACING=0 \
     229             :     SCOREP_TOTAL_MEMORY=1536000000 \
     230             :     SCOREP_FILTERING_FILE=$SPECTRE_HOME/support/Profiling/ScoreP/spectre.flt \
     231             :     mpirun -np 2 EvolveGhSingleBlackHole --input-file ./KerrSchild.yaml +ppn 91
     232             : ```
     233             : Depending on your profile, you might also need to increase the callpath depth by
     234             : setting
     235             : ```sh
     236             : SCOREP_PROFILING_MAX_CALLPATH_DEPTH=150
     237             : ```
     238             : Once you've successfully run the executable you should have a directory like
     239             : `scorep-20251006_1732_65976792745157592`. Please see the Score-P documentation
     240             : for how to analyze the output.