Line data Source code
1 0 : \cond NEVER
2 : Distributed under the MIT License.
3 : See LICENSE.txt for details.
4 : \endcond
5 : # %Running CCE {#tutorial_cce}
6 :
7 : \tableofcontents
8 :
9 : The basic instructions for getting up and running with a stand-alone
10 : CCE using external data are:
11 : - Clone spectre and build the CharacteristicExtract target
12 : - At this point (provided the build succeeds) you should have the
13 : executable `bin/CharacteristicExtract` in the build directory; You can now
14 : run it using an input file to provide options. The input file
15 : `tests/InputFiles/Cce/CharacteristicExtract.yaml` from the spectre
16 : source tree can help you get started with writing your input file.
17 : There are a few important notes there:
18 : - For resolution, the example input file has lmax (`Cce.LMax`) of 20, and
19 : filter lmax (`Filtering.FilterLMax`) of 18; that may run a bit slow for
20 : basic tests, but this value yields the best precision-to-run-time ratio
21 : for a typical BBH system. Note that precision doesn't improve above lmax 24,
22 : filter 22 (be sure to update the filter as you update lmax -- it should
23 : generally be around 2 lower than the maximum l to reduce possible aliasing).
24 : - If you want to just run through the end of the provided worldtube data,
25 : you can just omit the `EndTime` option and the executable will figure it
26 : out from the worldtube file.
27 : - The `ScriOutputDensity` adds extra interpolation points to the output,
28 : which is useful for finite-difference derivatives on the output data, but
29 : otherwise it'll just unnecessarily inflate the output files, so if you
30 : don't need the extra points, best just set it to 1.
31 : - If you're extracting at 100M or less (which isn't currently recommended due
32 : to the junk radiation being much worse), best to keep the `TargetStepSize`
33 : as .5 for 100M and perhaps even lower yet for nearer extraction.
34 : - The `InitializeJ` in the example file uses `ConformalFactor` which has been
35 : found to perform better than the other schemes implemented so far.
36 : Other schemes for `InitializeJ` can be found in namespace
37 : \link Cce::InitializeJ \endlink. Some of these are:
38 : - `ZeroNonSmooth`: Make `J` vanish
39 : - `NoIncomingRadiation`: Make \f$\Psi_0 = 0\f$; this does not actually lead
40 : to no incoming radiation, since \f$\Psi_0\f$ and \f$\Psi_4\f$ both include
41 : incoming and outgoing radiation.
42 : - `InverseCubic`: Ansatz where \f$J = A/r + B/r^3\f$
43 : - `ConformalFactor`: Try to make initial time coordinate as inertial as
44 : possible at \f$\mathscr{I}^+\f$.
45 :
46 : - An example of an appropriate submission command for slurm systems is:
47 : ```
48 : srun -n 1 -c 1 path/to/build/bin/CharacteristicExtract ++ppn 3 \
49 : --input-file path/to/input.yaml
50 : ```
51 : CCE doesn't currently scale to more than 4 cores, so those slurm options are
52 : best.
53 : - CCE will work faster if the input worldtube hdf5 file is chunked in small
54 : numbers of complete rows.
55 : This is relevant because by default, SpEC writes its worldtube files
56 : chunked along full time-series columns, which is efficient for
57 : compression, but not for reading in to SpECTRE -- in that case,
58 : it is recommended to rechunk the input file before running CCE
59 : for maximum performance. This can be done, for instance, using h5py
60 : (you will need to fill in filenames appropriate to your case in place
61 : of "BondiCceR0050.h5" and "RechunkBondiCceR0050.h5"):
62 : ```py
63 : import h5py
64 : input_file = "BondiCceR0050.h5"
65 : output_file = "RechunkBondiCceR0050.h5"
66 : with h5py.File(input_file,'r') as input_h5,\
67 : h5py.File(output_file, 'w') as output_h5:
68 : for dset in input_h5:
69 : if("Version" in dset):
70 : output_h5[dset] = input_h5[dset][()]
71 : continue
72 : number_of_columns = input_h5[dset][()].shape[1]
73 : output_h5.create_dataset(dset, data=input_h5[dset],
74 : maxshape=(None, number_of_columns),
75 : chunks=(4, number_of_columns), dtype='d')
76 : for attribute in input_h5[dset].attrs.keys():
77 : output_h5[dset].attrs[attribute] = input_h5[dset].attrs[attribute]
78 : ```
79 : - Note, these `*.h5` files required as an `input_file` (and also needed
80 : in the `.yaml` file) will need to be obtained from an SXS member who has
81 : completed a relevant `SpEC` run.
82 : - The output data will be written as spin-weighted spherical harmonic
83 : modes, one physical quantity per dataset, and each row will
84 : have the time value followed by the real and imaginary parts
85 : of the complex modes in m-varies-fastest order.
86 :
87 : Once you have the reduction data output file from a successful CCE run, you can
88 : confirm the integrity of the h5 file and its contents by running
89 : ```
90 : h5ls -r CharacteristicExtractReduction.h5/Cce.dir
91 : ```
92 :
93 : For the reduction file produced by a successful run, the output of the `h5ls`
94 : should resemble
95 : ```
96 : / Group
97 : /Cce Group
98 : /Cce/EthInertialRetardedTime.dat Dataset {3995/Inf, 163}
99 : /Cce/News.dat Dataset {3995/Inf, 163}
100 : /Cce/Psi0.dat Dataset {3995/Inf, 163}
101 : /Cce/Psi1.dat Dataset {3995/Inf, 163}
102 : /Cce/Psi2.dat Dataset {3995/Inf, 163}
103 : /Cce/Psi3.dat Dataset {3995/Inf, 163}
104 : /Cce/Psi4.dat Dataset {3995/Inf, 163}
105 : /Cce/Strain.dat Dataset {3995/Inf, 163}
106 : src.tar.gz Dataset {3750199}
107 : ```
108 :
109 : The `Strain.dat` represents the asymptotic transverse-traceless contribution
110 : to the metric scaled by the Bondi radius (to give the asymptotically leading
111 : part), the `News.dat` represents the first derivative of the strain, and each
112 : of the `Psi...` datasets represent the Weyl scalars, each scaled by the
113 : appropriate factor of the Bondi-Sachs radius to retrieve the asymptotically
114 : leading contribution.
115 :
116 : The `EthInertialRetardedTime.dat` is a diagnostic dataset that represents the
117 : angular derivative of the inertial retarded time, which determines the
118 : coordinate transformation that is performed at scri+.
119 :
120 : The following python script will load data from a successful CCE run and
121 : construct a plot of all of the physical waveform data.
122 : ```py
123 : import matplotlib.pyplot as plt
124 : import numpy as np
125 : import h5py as h5
126 :
127 :
128 : def spectre_real_mode_index(l, m):
129 : return 2 * (l**2 + l + m) + 1
130 :
131 :
132 : def spectre_imag_mode_index(l, m):
133 : return 2 * (l**2 + l + m) + 2
134 :
135 :
136 : def get_modes_from_block_output(filename, dataset, modes=[[2, 2], [3, 3]]):
137 : with h5.File(filename, "r") as h5_file:
138 : timeseries_data = (h5_file[dataset + ".dat"][()][:, [0] + list(
139 : np.array([[
140 : spectre_real_mode_index(x[0], x[1]),
141 : spectre_imag_mode_index(x[0], x[1])
142 : ] for x in modes]).flatten())])
143 : return timeseries_data
144 :
145 :
146 : plot_quantities = ["Strain", "News", "Psi0", "Psi1", "Psi2", "Psi3", "Psi4"]
147 : mode_set = [[2, 2], [3, 3]]
148 : filename = "CharacteristicExtractReduction.h5"
149 : output_plot_filename = "CCE_plot.pdf"
150 :
151 : legend = []
152 : for (mode_l, mode_m) in mode_set:
153 : legend = np.append(legend, [
154 : r"Re $Y_{" + str(mode_l) + r"\," + str(mode_m) + r"}$",
155 : r"Im $Y_{" + str(mode_l) + r"\," + str(mode_m) + r"}$"
156 : ])
157 :
158 : plt.figure(figsize=(8, 1.9 * len(plot_quantities)))
159 : for i in range(len(plot_quantities)):
160 : ax = plt.subplot(len(plot_quantities), 1, i + 1)
161 : timeseries = np.transpose(
162 : get_modes_from_block_output(
163 : filename, f"/Cce/{plot_quantities[i]}", mode_set
164 : )
165 : )
166 : for j in range(len(mode_set)):
167 : plt.plot(timeseries[0], timeseries[j + 1], linestyle='--', marker='')
168 : ax.set_xlabel("Simulation time (M)")
169 : ax.set_ylabel("Mode coefficient")
170 : ax.set_title(plot_quantities[i])
171 : plt.tight_layout()
172 : plt.savefig(output_plot_filename, dpi=400)
173 : plt.clf()
174 : ```
175 :
176 : ### Input data formats
177 :
178 : The worldtube data must be constructed as spheres of constant coordinate
179 : radius, and (for the time being) written to a filename of the format
180 : `...CceRXXXX.h5`, where the `XXXX` is to be replaced by the integer for which
181 : the extraction radius is equal to `XXXX`M. For instance, a 100M extraction
182 : should have filename `...CceR0100.h5`. This scheme of labeling files with the
183 : extraction radius is constructed for compatibility with SpEC worldtube data.
184 :
185 : There are two possible formats of the input data, one based on the Cauchy metric
186 : at finite radius, and one based on Bondi data. The metric data format must be
187 : provided as spherical harmonic modes with the following datasets:
188 : - `gxx.dat`, `gxy.dat`, `gxz.dat`, `gyy.dat`, `gyz.dat`, `gzz.dat`
189 : - `Drgxx.dat`, `Drgxy.dat`, `Drgxz.dat`, `Drgyy.dat`, `Drgyz.dat`, `Drgzz.dat`
190 : - `Dtgxx.dat`, `Dtgxy.dat`, `Dtgxz.dat`, `Dtgyy.dat`, `Dtgyz.dat`, `Dtgzz.dat`
191 : - `Shiftx.dat`, `Shifty.dat`, `Shiftz.dat`
192 : - `DrShiftx.dat`, `DrShifty.dat`, `DrShiftz.dat`
193 : - `DtShiftx.dat`, `DtShifty.dat`, `DtShiftz.dat`
194 : - `Lapse.dat`
195 : - `DrLapse.dat`
196 : - `DtLapse.dat`
197 :
198 : In this format, each row must start with the time stamp, and the remaining
199 : values are the complex modes in m-varies-fastest format. That is,
200 : ```
201 : "time", "Lapse_Re(0,0)", "Lapse_Im(0,0)",
202 : "Lapse_Re(1,1)", "Lapse_Im(1,1)", "Lapse_Re(1,0)", "Lapse_Im(1,0)",
203 : "Lapse_Re(1,-1)", "Lapse_Im(1,-1)",
204 : "Lapse_Re(2,2)", "Lapse_Im(2,2)", "Lapse_Re(2,1)", "Lapse_Im(2,1)",
205 : "Lapse_Re(2,0)", "Lapse_Im(2,0)", "Lapse_Re(2,-1)", "Lapse_Im(2,-1)",
206 : "Lapse_Re(2,-2)", "Lapse_Im(2,-2)"
207 : ```
208 : Each dataset in the file must also have an attribute named "Legend" which
209 : is an ASCII-encoded null-terminated variable-length string. That is, the HDF5
210 : type is:
211 : ```
212 : DATATYPE H5T_STRING {
213 : STRSIZE H5T_VARIABLE;
214 : STRPAD H5T_STR_NULLTERM;
215 : CSET H5T_CSET_ASCII;
216 : CTYPE H5T_C_S1;
217 : }
218 : ```
219 : This can be checked for a dataset by running
220 : ```
221 : h5dump -a DrLapse.dat/Legend CceR0150.h5
222 : ```
223 :
224 : The second format is Bondi-Sachs metric component data.
225 : This format is far more space-efficient (by around a factor of 4), and SpECTRE
226 : provides a separate executable for converting to the Bondi-Sachs worldtube
227 : format, `ReduceCceWorldtube`.
228 : The `ReduceCceWorldtube` executable should be run on a Cauchy metric worldtube
229 : file, and will produce a corresponding 'reduced' Bondi-Sachs worldtube file.
230 : The basic command-line arguments for the executable are:
231 : ```
232 : ReduceCceWorldtube --input-file CceR0050.h5 --output-file BondiCceR0050.h5\
233 : --lmax_factor 3
234 : ```
235 : The argument `--lmax_factor` determines the factor by which the resolution at
236 : which the boundary computation that is run will exceed the resolution of the
237 : input and output files.
238 : Empirically, we have found that `lmax_factor` of 3 is sufficient to achieve
239 : roundoff precision in all boundary data we have attempted, and an `lmax_factor`
240 : of 2 is usually sufficient to vastly exceed the precision of the simulation that
241 : provided the boundary dataset.
242 :
243 : The format is similar to the metric components, except in spin-weighted
244 : spherical harmonic modes, and the real (spin-weight-0) quantities omit the
245 : redundant negative-m modes and imaginary parts of m=0 modes.
246 : The quantities that must be provided by the Bondi-Sachs metric data format are:
247 : - `Beta.dat`
248 : - `DrJ.dat`
249 : - `DuR.dat`
250 : - `H.dat`
251 : - `J.dat`
252 : - `Q.dat`
253 : - `R.dat`
254 : - `U.dat`
255 : - `W.dat`
256 :
257 : The Bondi-Sachs data may also be used directly for CCE input data.
258 : To specify that the input type is in 'reduced' Bondi-Sachs form, use:
259 : ```
260 : ...
261 : Cce:
262 : H5IsBondiData: True
263 : ...
264 : ```
265 : Otherwise, for the Cauchy metric data format, use:
266 : ```
267 : ...
268 : Cce:
269 : H5IsBondiData: False
270 : ...
271 : ```
|
