SpECTRE Documentation Coverage Report
Current view: top level - __w/spectre/spectre/docs/Tutorials - CCE.md Hit Total Coverage
Commit: 2db722c93a8e9b106e406b439b79c8e05c2057fb Lines: 0 1 0.0 %
Date: 2021-03-03 22:01:00
Legend: Lines: hit not hit

          Line data    Source code
       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # %Running CCE {#tutorial_cce}
       6             : 
       7             : The basic instructions for getting up and running with a stand-alone
       8             : CCE using external data are:
       9             : - Clone spectre and build the CharacteristicExtract target
      10             : - At this point (provided the build succeeds) you should have the
      11             :   executable `bin/CharacteristicExtract` in the build directory; You can now
      12             :   run it using an input file to provide options. The input file
      13             :   `tests/InputFiles/Cce/CharacteristicExtract.yaml` from the spectre
      14             :   source tree can help you get started with writing your input file.
      15             :   There are a few important notes there:
      16             :   - For resolution, the example input file has lmax (`Cce.LMax`) of 12, and
      17             :     filter lmax (`Filtering.FilterLMax`) of 10; that'll run pretty fast but
      18             :     might be a little low for full precision. lmax 16, filter 14 should be
      19             :     pretty good, and typically precision doesn't improve above lmax 24,
      20             :     filter 22 (be sure to update the filter as you update lmax -- it should
      21             :     generally be around 2 lower than the maximum l to reduce possible aliasing).
      22             :   - If you want to just run through the end of the provided worldtube data,
      23             :     you can just omit the `EndTime` option and the executable will figure it
      24             :     out from the worldtube file.
      25             :   - The `ScriOutputDensity` adds extra interpolation points to the output,
      26             :     which is useful for finite-difference derivatives on the output data, but
      27             :     otherwise it'll just unnecessarily inflate the output files, so if you
      28             :     don't need the extra points, best just set it to 1.
      29             :   - If you're extracting at 100M or less, best to reduce the `TargetStepSize`,
      30             :     to around .5 at 100M and lower yet for nearer extraction.
      31             :   - The `InitializeJ` in the example file uses `InverseCubic` which is a pretty
      32             :     primitive scheme, but early tests indicate that it gives the best results
      33             :     for most systems.
      34             :     If initial data is a concern, you can also try replacing the `InverseCubic`
      35             :     entry with:
      36             :   ```
      37             :     NoIncomingRadiation:
      38             :       AngularCoordTolerance: 1.0e-13
      39             :       MaxIterations: 500
      40             :       RequireConvergence: true
      41             :   ```
      42             :   which are probably pretty good choices for those parameters,
      43             :   and the `RequireConvergence: true` will cause the iterative solve in
      44             :   this version to error out if it doesn't find a good frame.
      45             : 
      46             : - An example of an appropriate submission command for slurm systems is:
      47             :   ```
      48             :   srun -n 1 -c 1 path/to/build/bin/CharacteristicExtract ++ppn 3 \
      49             :  --input-file path/to/input.yaml
      50             :   ```
      51             :   CCE doesn't currently scale to more than 4 cores, so those slurm options are
      52             :   best.
      53             : - CCE will work faster if the input worldtube hdf5 file is chunked in small
      54             :   numbers of complete rows.
      55             :   This is relevant because by default, SpEC writes its worldtube files
      56             :   chunked along full time-series columns, which is efficient for
      57             :   compression, but not for reading in to SpECTRE -- in that case,
      58             :   it is recommended to rechunk the input file before running CCE
      59             :   for maximum performance. This can be done, for instance, using h5py
      60             :   (you will need to fill in filenames appropriate to your case in place
      61             :   of "BondiCceR0050.h5" and "RechunkBondiCceR0050.h5"):
      62             :   ```py
      63             :   import h5py
      64             :   input_file = "BondiCceR0050.h5"
      65             :   output_file = "RechunkBondiCceR0050.h5"
      66             :   with h5py.File(input_file,'r') as input_h5,\
      67             :     h5py.File(output_file, 'w') as output_h5:
      68             :     for dset in input_h5:
      69             :         if("Version" in dset):
      70             :             output_h5[dset] = input_h5[dset][()]
      71             :             continue
      72             :         number_of_columns = input_h5[dset][()].shape[1]
      73             :         output_h5.create_dataset(dset, data=input_h5[dset],
      74             :                                  maxshape=(None, number_of_columns),
      75             :                                  chunks=(4, number_of_columns), dtype='d')
      76             :         for attribute in input_h5[dset].attrs.keys():
      77             :             output_h5[dset].attrs[attribute] = input_h5[dset].attrs[attribute]
      78             :   ```
      79             : - The output data will be written as spin-weighted spherical harmonic
      80             :   modes, one physical quantity per dataset, and each row will
      81             :   have the time value followed by the real and imaginary parts
      82             :   of the complex modes in m-varies-fastest order.
      83             : 
      84             : Once you have the volume data output file from a successful CCE run, you can
      85             : confirm the integrity of the h5 file and its contents by running
      86             : ```
      87             : h5ls CharacteristicExtractVolumeData0.h5
      88             : ```
      89             : 
      90             : For the volume file produced by a successful run, the output of the `h5ls`
      91             : should resemble
      92             : ```
      93             : EthInertialRetardedTime.dat Dataset {3995/Inf, 163}
      94             : News.dat                 Dataset {3995/Inf, 163}
      95             : Psi0.dat                 Dataset {3995/Inf, 163}
      96             : Psi1.dat                 Dataset {3995/Inf, 163}
      97             : Psi2.dat                 Dataset {3995/Inf, 163}
      98             : Psi3.dat                 Dataset {3995/Inf, 163}
      99             : Psi4.dat                 Dataset {3995/Inf, 163}
     100             : Strain.dat               Dataset {3995/Inf, 163}
     101             : src.tar.gz               Dataset {3750199}
     102             : ```
     103             : 
     104             : The `Strain.dat` represents the asymptotic transverse-traceless contribution
     105             : to the metric scaled by the Bondi radius (to give the asymptotically leading
     106             : part), the `News.dat` represents the first derivative of the strain, and each
     107             : of the `Psi...` datasets represent the Weyl scalars, each scaled by the
     108             : appropriate factor of the Bondi-Sachs radius to retrieve the asymptotically
     109             : leading contribution.
     110             : 
     111             : The `EthInertialRetardedTime.dat` is a diagnostic dataset that represents the
     112             : angular derivative of the inertial retarded time, which determines the
     113             : coordinate transformation that is performed at scri+.
     114             : 
     115             : The following python script will load data from a successful CCE run and
     116             : construct a plot of all of the physical waveform data.
     117             : ```py
     118             : import matplotlib.pyplot as plt
     119             : import numpy as np
     120             : import h5py as h5
     121             : 
     122             : 
     123             : def spectre_real_mode_index(l, m):
     124             :     return 2 * (l**2 + l + m)
     125             : 
     126             : 
     127             : def spectre_imag_mode_index(l, m):
     128             :     return 2 * (l**2 + l + m) + 1
     129             : 
     130             : 
     131             : def get_modes_from_block_output(filename, dataset, modes=[[2, 2], [3, 3]]):
     132             :     with h5.File(filename, "r") as h5_file:
     133             :         timeseries_data = (h5_file[dataset + ".dat"][()][:, [0] + list(
     134             :             np.array([[
     135             :                 spectre_real_mode_index(x[0], x[1]),
     136             :                 spectre_imag_mode_index(x[0], x[1])
     137             :             ] for x in modes]).flatten())])
     138             :     return timeseries_data
     139             : 
     140             : 
     141             : plot_quantities = ["Strain", "News", "Psi0", "Psi1", "Psi2", "Psi3", "Psi4"]
     142             : mode_set = [[2, 2], [3, 3]]
     143             : filename = "CharacteristicExtractVolume0.h5"
     144             : output_plot_filename = "CCE_plot.pdf"
     145             : 
     146             : legend = []
     147             : for (mode_l, mode_m) in mode_set:
     148             :     legend = np.append(legend, [
     149             :         r"Re $Y_{" + str(mode_l) + r"\," + str(mode_m) + r"}$",
     150             :         r"Im $Y_{" + str(mode_l) + r"\," + str(mode_m) + r"}$"
     151             :     ])
     152             : 
     153             : plt.figure(figsize=(8, 1.9 * len(plot_quantities)))
     154             : for i in range(len(plot_quantities)):
     155             :     ax = plt.subplot(len(plot_quantities), 1, i + 1)
     156             :     timeseries = np.transpose(
     157             :         get_modes_from_block_output(filename, plot_quantities[i], mode_set))
     158             :     for j in range(len(mode_set)):
     159             :         plt.plot(timeseries[0], timeseries[j + 1], linestyle='--', marker='')
     160             :     ax.set_xlabel("Simulation time (M)")
     161             :     ax.set_ylabel("Mode coefficient")
     162             :     ax.set_title(plot_quantities[i])
     163             : plt.tight_layout()
     164             : plt.savefig(output_plot_filename, dpi=400)
     165             : plt.clf()
     166             : ```
     167             : 
     168             : ### Input data formats
     169             : 
     170             : The worldtube data must be constructed as spheres of constant coordinate
     171             : radius, and (for the time being) written to a filename of the format
     172             : `...CceRXXXX.h5`, where the `XXXX` is to be replaced by the integer for which
     173             : the extraction radius is equal to `XXXX`M. For instance, a 100M extraction
     174             : should have filename `...CceR0100.h5`. This scheme of labeling files with the
     175             : extraction radius is constructed for compatibility with SpEC worldtube data.
     176             : 
     177             : There are two possible formats of the input data, one based on the Cauchy metric
     178             : at finite radius, and one based on Bondi data. The metric data format must be
     179             : provided as spherical harmonic modes with the following datasets:
     180             : - `gxx.dat`, `gxy.dat`, `gxz.dat`, `gyy.dat`, `gyz.dat`, `gzz.dat`
     181             : - `Drgxx.dat`, `Drgxy.dat`, `Drgxz.dat`, `Drgyy.dat`, `Drgyz.dat`, `Drgzz.dat`
     182             : - `Dtgxx.dat`, `Dtgxy.dat`, `Dtgxz.dat`, `Dtgyy.dat`, `Dtgyz.dat`, `Dtgzz.dat`
     183             : - `Shiftx.dat`, `Shifty.dat`, `Shiftz.dat`
     184             : - `DrShiftx.dat`, `DrShifty.dat`, `DrShiftz.dat`
     185             : - `DtShiftx.dat`, `DtShifty.dat`, `DtShiftz.dat`
     186             : - `Lapse.dat`
     187             : - `DrLapse.dat`
     188             : - `DtLapse.dat`
     189             : 
     190             : In this format, each row must start with the time stamp, and the remaining
     191             : values are the complex modes in m-varies-fastest format.
     192             : 
     193             : The second format is Bondi-Sachs metric component data.
     194             : This format is far more space-efficient (by around a factor of 4), and SpECTRE
     195             : provides a separate executable for converting to the Bondi-Sachs worldtube
     196             : format, `ReduceCceWorldtube`.
     197             : The `ReduceCceWorldtube` executable should be run on a Cauchy metric worldtube
     198             : file, and will produce a corresponding 'reduced' Bondi-Sachs worldtube file.
     199             : The basic command-line arguments for the executable are:
     200             : ```
     201             : ReduceCceWorldtube --input-file CceR0050.h5 --output-file BondiCceR0050.h5\
     202             :  --lmax_factor 3
     203             : ```
     204             : The argument `--lmax_factor` determines the factor by which the resolution at
     205             : which the boundary computation that is run will exceed the resolution of the
     206             : input and output files.
     207             : Empirically, we have found that `lmax_factor` of 3 is sufficient to achieve
     208             : roundoff precision in all boundary data we have attempted, and an `lmax_factor`
     209             : of 2 is usually sufficient to vastly exceed the precision of the simulation that
     210             : provided the boundary dataset.
     211             : 
     212             : The format is similar to the metric components, except in spin-weighted
     213             : spherical harmonic modes, and the real (spin-weight-0) quantities omit the
     214             : redundant negative-m modes and imaginary parts of m=0 modes.
     215             : The quantities that must be provided by the Bondi-Sachs metric data format are:
     216             : - `Beta.dat`
     217             : - `DrJ.dat`
     218             : - `DuR.dat`
     219             : - `H.dat`
     220             : - `J.dat`
     221             : - `Q.dat`
     222             : - `R.dat`
     223             : - `U.dat`
     224             : - `W.dat`
     225             : 
     226             : The Bondi-Sachs data may also be used directly for CCE input data.
     227             : To specify that the input type is in 'reduced' Bondi-Sachs form, use:
     228             : ```
     229             : ...
     230             : Cce:
     231             :   H5IsBondiData: True
     232             : ...
     233             : ```
     234             : Otherwise, for the Cauchy metric data format, use:
     235             : ```
     236             : ...
     237             : Cce:
     238             :   H5IsBondiData: False
     239             : ...
     240             : ```

Generated by: LCOV version 1.14