spectre.support.DirectoryStructure¶

Functions

`list_checkpoints`(checkpoints_dir)	All checkpoints in the 'checkpoints_dir'
`list_pipeline_steps`(base_dir)	List all subdirectories in the base directory
`list_segments`(segments_dir)	All segments in the 'segments_dir'

Classes

`Checkpoint`(path, id)	State of a simulation saved to disk
`PipelineStep`(path, id, label)	A step in a pipeline
`Segment`(path, id)	Part of a simulation that ran as one executable invocation

class spectre.support.DirectoryStructure.Checkpoint(path: Path, id: int)¶

State of a simulation saved to disk

An executable can write multiple checkpoints during its execution, and can be restarted from those checkpoints. The ‘id’ enumerates the checkpoints.

We currently write checkpoints in a directory structure like this:

``` RUN_DIR/

Checkpoints/
Checkpoint_0000/ Checkpoint_0001/ …

```

WARNING: Don’t assume checkpoints always exist in the above directory structure. You don’t want your code to break when checkpoints are copied, moved around, or renamed.

NAME_PATTERN = re.compile('Checkpoint_(\\d+)')¶

NUM_DIGITS = 4¶

id: int¶

classmethod match(path: str | Path) → Checkpoint | None¶: Checks if the ‘path’ is a checkpoint

path: Path¶

class spectre.support.DirectoryStructure.PipelineStep(path: Path, id: int, label: str)¶

A step in a pipeline

Each pipeline step is a numbered directory with a label, e.g. “000_InitialData”. It can contain a simulation or a pre- or post-processing step such as archiving. Here is an example for the directory structure:

``` BASE_DIR/

000_InitialData/
InputFile.yaml Submit.sh …

001_Inspiral/

Segment_0000/
InputFile.yaml Submit.sh …

Segment_0001/
…

002_InitialData/
…

…

```

Note: “InitialData” and “Inspiral” are examples, any name can be used.

NAME_PATTERN = re.compile('(\\d+)_(.+)')¶

NUM_DIGITS = 3¶

classmethod first(directory: str | Path, label: str) → PipelineStep¶: Create the first directory in a sequence

id: int¶

label: str¶

classmethod match(path: str | Path) → PipelineStep | None¶: Checks if the ‘path’ is a step in the pipeline

next(label: str) → PipelineStep¶: Get the next directory in the sequence

path: Path¶

class spectre.support.DirectoryStructure.Segment(path: Path, id: int)¶

Part of a simulation that ran as one executable invocation

We have to split simulations into segments because supercomputers don’t typically support unlimited run times. Therefore, we terminate the job, write the simulation state to disk as a checkpoint, and submit a new job that restarts from the last checkpoint.

We currently write segments in a directory structure like this:

``` SEGMENTS_DIR/

Segment_0000/
InputFile.yaml Submit.sh Output.h5 Checkpoints/

Segment_0001/…

```

WARNING: Don’t assume that simulations always have the above directory structure. You don’t want your code to break when files are copied, moved around, or renamed. Instead of relying on some directory structure, have your code take the files it needs as input. This is quite easy using globs.

NAME_PATTERN = re.compile('Segment_(\\d+)')¶

NUM_DIGITS = 4¶

property checkpoints: List[Checkpoint]¶

property checkpoints_dir: Path¶

classmethod first(directory: str | Path) → Segment¶

id: int¶

classmethod match(path: str | Path) → Segment | None¶: Checks if the ‘path’ is a segment

property next: Segment¶

path: Path¶

spectre.support.DirectoryStructure.list_checkpoints(checkpoints_dir: str | Path) → List[Checkpoint]¶: All checkpoints in the ‘checkpoints_dir’

spectre.support.DirectoryStructure.list_pipeline_steps(base_dir: str | Path) → List[PipelineStep]¶: List all subdirectories in the base directory

spectre.support.DirectoryStructure.list_segments(segments_dir: str | Path) → List[Segment]¶: All segments in the ‘segments_dir’