Line data    Source code

       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # Installation on Clusters {#installation_on_clusters}
       6             : 
       7             : \tableofcontents
       8             : 
       9             : The installation instructions are the same for most systems because we use shell
      10             : scripts to set up the environment for each supercomputer. We describe the
      11             : generic installation instructions once, and only note special instructions if
      12             : necessary for the particular system. If you have already built SpECTRE and just
      13             : want to load the modules, source the shell file for your system and run
      14             : `spectre_load_modules`.
      15             : 
      16             : \note Sample submit scripts for some systems are available in
      17             : `support/SubmitScripts`.
      18             : 
      19             : ## General Instructions
      20             : 
      21             : 1. Run `export SPECTRE_HOME=/path/to/where/you/want/to/clone`
      22             : 2. Clone SpECTRE using `git clone SPECTRE_URL $SPECTRE_HOME` where the
      23             :    `SPECTRE_URL` is the GitHub url you want to use to clone the repo.
      24             : 3. Run `cd $SPECTRE_HOME && mkdir build && cd build`
      25             : 4. Run `. $SPECTRE_HOME/support/Environments/SYSTEM_TO_RUN_ON_gcc.sh`, where
      26             :    `SYSTEM_TO_RUN_ON` is replaced by the name of the system as described in the
      27             :    relevant section below.
      28             : 5. If you haven't already installed the dependencies, run
      29             :    `export SPECTRE_DEPS=/path/to/where/you/want/the/deps`
      30             :    Then run `spectre_setup_modules $SPECTRE_DEPS`. This
      31             :    will take a while to finish. Near the end the command will tell you how to
      32             :    make the modules available by providing a `module use` command. Make
      33             :    sure you are providing an absolute path to `spectre_setup_modules`.
      34             : 6. Run `module use $SPECTRE_DEPS/modules`
      35             : 7. Run `spectre_run_cmake`, if you get module loading errors run
      36             :    `spectre_unload_modules` and try running `spectre_run_cmake` again. CMake
      37             :    should set up successfully.
      38             : 8. Build the targets you are interested in by running, e.g.
      39             :    `make -j4 test-executables`
      40             : 
      41             : ## Anvil at Purdue University
      42             : 
      43             : You should build and run tests on a compute node. You can get a compute node by
      44             : running
      45             : ```
      46             : sinteractive -N1 -n 20 -p debug -t 60:00
      47             : ```
      48             : Avoid running `module purge` because this also removes various default modules
      49             : that are necessary for proper operation. Instead, use `module
      50             : restore`. Currently the tests can only be run in serial, e.g. `ctest -j1`
      51             : because all the MPI jobs end up being launched on the same core.
      52             : 
      53             : ## Frontera at TACC
      54             : 
      55             : Follow the general instructions, using `frontera` for `SYSTEM_TO_RUN_ON`.
      56             : 
      57             : Processes running on the head nodes have restrictions on memory use
      58             : that will prevent linking the main executables.  It is better to
      59             : compile on an interactive node.  Interactive nodes can be requested
      60             : with the `idev` command.
      61             : 
      62             : For unknown reasons, incremental builds work poorly on frontera.
      63             : Running `make` will often unnecessarily recompile SpECTRE libraries.
      64             : 
      65             : ## CaltechHPC at Caltech
      66             : 
      67             : Cluster documentation: https://www.hpc.caltech.edu
      68             : 
      69             : Follow the general instructions, using `caltech_hpc` for `SYSTEM_TO_RUN_ON`,
      70             : except you don't need to install any dependencies, so you can skip step 5 and
      71             : step 6.
      72             : 
      73             : There are multiple types of compute nodes on CaltechHPC, which are listed on
      74             : https://www.hpc.caltech.edu/resources. We compile SpECTRE to be compatible with
      75             : all of them.
      76             : 
      77             : When you go to build, you will need to get an interactive node (login nodes
      78             : limit the amount of memory accessible to individual users, to below the amount
      79             : necessary to build SpECTRE). Slurm commands are listed on
      80             : https://www.hpc.caltech.edu/documentation/slurm-commands. For example to ensure
      81             : you get an entire node to build on, use the following command:
      82             : 
      83             : ```
      84             : srun [--reservation=sxs_standing] [--constraint=skylake/cascadelake/icelake] \
      85             :   -t 02:00:00 -N 1 --exclusive -D . --pty /bin/bash
      86             : ```
      87             : 
      88             : If you are part of the SXS collaboration, you can use our reserved nodes by
      89             : specifying `--reservation=sxs_standing`. Our reserved nodes are currently
      90             : cascadelake nodes with 56 cores each. If you want to use another type of node
      91             : you can specify a `--constraint`. However, note that the reservation flag won't
      92             : work for nodes other than cascadelake.
      93             : 
      94             : Be sure to re-source the correct environment files once you get the interactive
      95             : node shell.
      96             : 
      97             : ## Ocean at Fullerton
      98             : 
      99             : Follow the general instructions, using `ocean` for `SYSTEM_TO_RUN_ON`,
     100             : you do not need to install any dependencies, so you can skip steps 5 and 6.
     101             : 
     102             : ## Mbot at Cornell
     103             : 
     104             : Follow steps 1-3 of the general instructions.
     105             : 
     106             : The only modules you need to load on `mbot` are `gcc/11.4.0 spectre-deps`.
     107             : This  will load everything you need, including LLVM/Clang. You
     108             : can source the environment file by running
     109             : `. $SPECTRE_HOME/support/Environments/mbot.sh` and load the modules using
     110             : `spectre_load_modules`. This offers two functions, `spectre_run_cmake_gcc`
     111             : and `spectre_run_cmake_clang` for the different compilers.
     112             : These both default to `Release` mode. If you are developing code, please use
     113             : either `spectre_run_cmake_clang -D CMAKE_BUILD_TYPE=Debug` or
     114             : `spectre_run_cmake_clang -D SPECTRE_DEBUG=ON` (you can also use gcc).
     115             : The second command with `SPECTRE_DEBUG=ON` enables sanity checks and
     116             : optimizations. This means it can be used in production-level runs to ensure
     117             : there aren't any subtle bugs that might only arise after a decently
     118             : long simulation.
     119             : 
     120             : ## Sonic at ICTS-TIFR
     121             : 
     122             : Cluster documentation: https://it.icts.res.in/docs/sonic-cluster
     123             : 
     124             : 1. Clone the spectre source code and set `SPECTRE_HOME` to point it.
     125             : 2. Create a directory for building SpECTRE (`SPECTRE_BUILD`).
     126             : 3. Setup the software environment
     127             :     ```
     128             :     . /mnt/pfs/vaishak.p/soft/modules-5.2.0/init/bash
     129             :     ```
     130             :     Add these lines to your `.bashrc` and source it
     131             :     if you would like to make it persistent.
     132             : 4. Source the `$SPECTRE_HOME/support/Environments/Sonic.sh` script.
     133             : 5. Build SpECTRE by changing to `$SPECTRE_BUILD` and using
     134             :    one of the variants of the `spectre_run_cmake` commands.
     135             :    This by default compiles SpECTRE in Release mode,
     136             :    without debug symbols and LTO, and with system malloc.
     137             :    If you wish to compile in debug mode, with LTO/JEMALLOC, or other options,
     138             :    simply suffix the additional cmake flags to the command.
     139             : 6. Use `make` to build your targets.