Line data    Source code

       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # %Setting up checkpoints and restarts {#tutorial_checkpoint_restart}
       6             : 
       7             : \tableofcontents
       8             : 
       9             : SpECTRE executables can write checkpoints that save their instantaneous state to
      10             : disc; the execution can be restarted later from a saved checkpoint. This feature
      11             : is useful for expensive simulations that would run longer than the wallclock
      12             : limits on a supercomputer system.
      13             : 
      14             : Executables can checkpoint when:
      15             : 1. The `default_phase_order` member variable in the `Metavariables` includes a
      16             :    `WriteCheckpoint` phase.
      17             : 2. The `WriteCheckpoint` phase is run by a `PhaseControl` specified in the
      18             :    `Metavariables` and the input file. The two supported ways of running the
      19             :    checkpoint phase are:
      20             :    - with `CheckpointAndExitAfterWallclock`. This is the recommended phase
      21             :      control for checkpointing, because it writes only one checkpoint before
      22             :      cleanly terminating the code.
      23             :      This reduces the disc space taken up by checkpoint files and stops using
      24             :      up the allocation's CPU-hours on work that would be redone anyway after the
      25             :      run is restarted.
      26             :      The executable will return exit code 2 when it terminates from
      27             :      `CheckpointAndExitAfterWallclock`, meaning it is incomplete and should
      28             :      continue from the checkpoint. See `Parallel::ExitCode` for a definition of
      29             :      all exit code.
      30             :    - using `VisitAndReturn(WriteCheckpoint)`. This is useful for writing more
      31             :      frequent checkpoint files, which could help when debugging a run by
      32             :      restarting it from just before the failure.
      33             : 
      34             : To restart an executable from a checkpoint file, run a command like this:
      35             : ```
      36             : ./MySpectreExecutable +restart Checkpoints/Checkpoint_0123
      37             : ```
      38             : where the `0123` should be the number of the checkpoint to restart from. You can
      39             : also use the \ref tutorial_cli "command-line interface (CLI)" for restarting:
      40             : ```
      41             : ./spectre run INPUT_FILE --from-last-checkpoint Checkpoints/
      42             : ```
      43             : 
      44             : There are a number of caveats in the current implementation of checkpointing
      45             : and restarting:
      46             : 
      47             : 1. The same binary must be used when writing the checkpoint and when restarting
      48             :    from the checkpoint. If a different binary is used to restart the code,
      49             :    there are no guarantees that the code will restart or that the continued
      50             :    execution will be correct.
      51             : 2. The code must be restarted on the same hardware configuration used when
      52             :    writing the checkpoint --- this means the same number of nodes with the same
      53             :    number of processors per node.
      54             : 3. When using `CheckpointAndExitAfterWallclock` to trigger checkpoints, note
      55             :    that the elapsed wallclock time is checked only when the `PhaseControl` is
      56             :    run, i.e., at global synchronization points defined in the input file.
      57             :    This means that to write a checkpoint in the 30 minutes before the end of a
      58             :    job's queue time, the triggers in the input file must trigger global
      59             :    synchronizations at least once every 30 minutes (and probably 2-3 times so
      60             :    there is a margin for the time to write files to disc, etc). It is currently
      61             :    up to the user to find the balance between too-frequent synchronizations
      62             :    (that slow the code) and too-infrequent synchronizations (that won't allow
      63             :    checkpoints to be written).
      64             : 
      65             : Certain simulation parameters can be modified when restarting from a checkpoint
      66             : file. This is done by parsing a new input file containing just those options to
      67             : modify; all other options will preserve their value from the original run.
      68             : 
      69             : Note, however, that not all tags are permitted to be modified: in the current
      70             : implementation, only tags from the `const_global_cache_tags` that also have a
      71             : member variable `static constexpr bool is_overlayable = true;` can be modified.
      72             : The reason for this "opt-in" design is that in general, most tags interact with
      73             : past or current simulation data in a way that would invalidate the simulation
      74             : state if the tag were modified on restart (example: changing the domain
      75             : invalidates all spatial data, changing the timestepper invalidates the history).
      76             : Only tags that do not interact with the state should be permitted to be updated.
      77             : For example: activation thresholds on various algorithms, or frequency of data
      78             : observation, are safe parameters to modify.
      79             : 
      80             : The executable will update the global cache with new input file values during
      81             : the phase `UpdateOptionsAtRestartFromCheckpoint`. The restart logic
      82             : automatically directs code flow to this phase after a restart.
      83             : 
      84             : In this option-updating phase, the code tries to read an "overlay" input file
      85             : whose name is computed from the original input file and the number of the
      86             : checkpoint used to restart. Say the original input file is `path/to/Input.yaml`
      87             : and the code is restarted using a checkpoint
      88             : `+restart Checkpoints/Checkpoint_0123`, then the overlay input file to read
      89             : has name `path/to/Input.overlay_0123.yaml`. If this file does not exist, the
      90             : executable continues with previous parameter values.
      91             : 
      92             : After the `UpdateOptionsAtRestartFromCheckpoint` phase runs, the code will
      93             : automatically transition to the `Restart` phase.  This phase is usually used to
      94             : redo various registration actions that may have been invalidated by the restart
      95             : or option-overlaying.  Control is then returned to the normal phase-arbitration
      96             : code, which will usually return to the phase that was active before the
      97             : checkpoint.