Line data Source code
1 0 : \cond NEVER
2 : Distributed under the MIT License.
3 : See LICENSE.txt for details.
4 : \endcond
5 : # %Running CCE {#tutorial_cce}
6 :
7 : \tableofcontents
8 :
9 : ## Acquiring the CCE module {#acquiring_the_cce_module}
10 :
11 : There are a couple different ways to acquire the CCE module/executable.
12 :
13 : ### From a release {#cce_from_release}
14 :
15 : Starting from late May 2024, in every
16 : [Release of SpECTRE](https://github.com/sxs-collaboration/spectre/releases) we
17 : offer a tarball that contains everything needed to run CCE on a large number of
18 : different systems. This will be under the `Assets` section towards the bottom
19 : of the release (there may be a lot of text detailing what's been updated in this
20 : release). Inside this tarball is
21 :
22 : - the CCE executable `CharacteristicExtract`
23 : - an example YAML input file
24 : - an example set of Bondi-Sachs worldtube data in the `Tests/` directory (see
25 : [Input worldtube data formats](#input_worldtube_data_format) section)
26 : - example output from CCE in the `Tests/` directory
27 : - a `PreprocessCceWorldtube` executable and YAML files for converting between
28 : [worldtube data formats](#input_worldtube_data_format) in the
29 : `PreprocessCceWorldtube/` directory
30 : - a `WriteCceWorldtubeCoordsToFile` executable that writes
31 : [grid points on a sphere](#spherical_nodes) to a text file in the
32 : `PreprocessCceWorldtube/` directory
33 : - a python script `CheckCceOutput.py` (meant to be run from the root of the
34 : tarball and after you run the example YAML input file also in the root of the
35 : tarball) that will check if the example output is correct
36 :
37 : \note The tarball is `.xz` so use `tar -xf TarName.tar.xz` to extract. The `-z`
38 : flag to use gzip will cause an error.
39 :
40 : See [Running the CCE executable](#running_the_cce_executable) for how to run
41 : CCE.
42 :
43 : We have tested that this executable works natively on the following machines:
44 :
45 : - Expanse
46 : - Anvil
47 : - Stampede3
48 : - Delta (if you add
49 : `LD_LIBRARY_PATH=/sw/spack/deltas11-2023-03/apps/linux-rhel8-x86_64/gcc-8.5.0/gcc-11.4.0-yycklku/lib64/:$LD_LIBRARY_PATH`
50 : before running CCE)
51 : - Perlmutter
52 : - Ubuntu 18.04 LTS or later (LTS version only)
53 :
54 : We have also tested that this executable works inside our
55 : [`dev` Docker container](https://hub.docker.com/r/sxscollaboration/spectre/tags)
56 : on the following machines (in addition to the ones above):
57 :
58 : - Frontera
59 : - Delta
60 :
61 : ### From Docker
62 :
63 : You can download a docker image `sxscollaboration/spectre:deploy` which has a
64 : few pre-built executables within, including the ones listed above in the
65 : [release](#cce_from_release) section. See the containerized releases section of
66 : our \ref installation instructions for how start the container.
67 :
68 : The input files can be found within the container at
69 : `/work/spectre/tests/InputFiles/`.
70 :
71 : ### From source
72 :
73 : You can clone the [spectre repo](https://github.com/sxs-collaboration/spectre)
74 : and follow the instructions on the \ref installation page to obtain an
75 : environment to configure and build SpECTRE. Once you have a configured `build`
76 : directory, build the CCE executable with
77 :
78 : ```
79 : make CharacteristicExtract
80 : ```
81 :
82 : \note You may want to add the `-j4` flag to speed up compilation. However, be
83 : warned that this executable will need several GB of memory to build.
84 :
85 : ## Input worldtube data format {#input_worldtube_data_format}
86 :
87 : In order to run the CCE executable, the worldtube data must be represented as
88 : Bondi-Sachs variables decomposed as a subset of spin-weighted spherical harmonic
89 : modes on a sphere of constant coordinate radius. We have chosen this format
90 : because it is far more space-efficient to store on disk than other formats. This
91 : section will detail the
92 : [required data format](#required_h5_worldtube_data_format), provide options for
93 : [converting worldtube data](#converting_worldtube_data) from other NR codes into
94 : our format, and give insights into
95 : [what the worldtube data should look like](#worldtube_data_looks).
96 :
97 : ### Required H5 worldtube data format {#required_h5_worldtube_data_format}
98 :
99 : Within the H5 file that holds the worldtube data, there must be the following
100 : datasets with these exact names (including the `.dat` suffix):
101 :
102 : - `Beta.dat`
103 : - `DrJ.dat`
104 : - `DuR.dat`
105 : - `H.dat`
106 : - `J.dat`
107 : - `Q.dat`
108 : - `R.dat`
109 : - `U.dat`
110 : - `W.dat`
111 :
112 : Each dataset in the file must also have an attribute named `Legend` which
113 : is an ASCII-encoded null-terminated variable-length string. That is, the HDF5
114 : type is:
115 :
116 : ```
117 : DATATYPE H5T_STRING {
118 : STRSIZE H5T_VARIABLE;
119 : STRPAD H5T_STR_NULLTERM;
120 : CSET H5T_CSET_ASCII;
121 : CTYPE H5T_C_S1;
122 : }
123 : ```
124 :
125 : This can be checked for a dataset by running
126 :
127 : ```
128 : h5dump -a Beta.dat/Legend WorldtubeFile.h5
129 : ```
130 :
131 : For the ordering of the data, we use spherical harmonic conventions documented
132 : by the ylm::Spherepack class. Each row must start with the time stamp, and the
133 : remaining values are the complex modes in m-varies-fastest format. For
134 : spin-weight zero Bondi variables (`Beta`, `R`, `DuR`, `W`), we omit the
135 : redundant negative-m modes and imaginary parts of the m=0 modes to save space on
136 : disk. Here is an example of a legend for the spin-weight zero variables:
137 :
138 : ```
139 : "time", "Re(0,0)", "Re(1,0)", "Re(1,1)", "Im(1,1)", "Re(2,0)",
140 : "Re(2,1)", "Im(2,1)", "Re(2,2)", "Im(2,2)", "Re(3,0)", "Re(3,1)",
141 : "Im(3,1)", "Re(3,2)", "Im(3,2)", "Re(3,3)", "Im(3,3)", ...
142 : ```
143 :
144 : For non-zero spin-weight Bondi variables (`J`, `DrJ`, `H`, `Q`, `U`) we must
145 : store all complex m-modes. Here is an example of a legend for variables where
146 : all complex m-modes must be specified:
147 :
148 : ```
149 : "time", "Re(0,0)", "Im(0,0)", "Re(1,-1)", "Im(1,-1)", "Re(1,0)", "Im(1,0)",
150 : "Re(1,1)", "Im(1,1)", "Re(2,-2)", "Im(2,-2)", "Re(2,-1)", "Im(2,-1)", "Re(2,0)",
151 : "Im(2,0)", "Re(2,1)", "Im(2,1)", "Re(2,2)", "Im(2,2)", ...
152 : ```
153 :
154 : We don't have strict requirement on the name of the H5 file that holds the
155 : worldtube data. However, it is recommended to name the H5 file `...CceRXXXX.h5`,
156 : where the `XXXX` is to be replaced by the zero-padded integer for which the
157 : extraction radius is equal to `XXXX`M. For instance, a 100M extraction should
158 : have filename `...CceR0100.h5`. If you do not adhere to this naming convention,
159 : you will need to specify the extraction radius in your YAML input file.
160 :
161 : \note This scheme of labeling files with the extraction radius is constructed
162 : for compatibility with worldtube data from the SXS Collaboration's SpEC code.
163 :
164 : ### Converting to the required H5 format {#converting_worldtube_data}
165 :
166 : Unless you are using worldtube data that was generated from SpECTRE (or SpEC),
167 : it's possible that your worldtube data is not in the correct format. We allow
168 : conversion into our data format from a few other data formats using the
169 : [`PreprocessCceWorldtube` executable provided](#acquiring_the_cce_module). These are
170 :
171 : - Nodal cartesian metric data (which we refer to as "metric nodal")
172 : - Modal cartesian metric data (which we refer to as "metric modal")
173 : - Nodal Bondi-Sachs data (which we refer to as "bondi nodal")
174 :
175 : Requirements for these data formats are listed below.
176 :
177 : #### Spherical harmonic modes {#spherical_modes}
178 :
179 : When we refer to a "modal" data format, we mean that the worldtube data are
180 : stored as spherical harmonic coefficients (a.k.a. modes). We use spherical
181 : harmonic conventions documented by the ylm::Spherepack class. For each dataset,
182 : each row must start with the time stamp, and the remaining values are the
183 : complex modes in m-varies-fastest format. That is,
184 :
185 : ```
186 : "time", "Re(0,0)", "Im(0,0)", "Re(1,-1)", "Im(1,-1)", "Re(1,0)", "Im(1,0)",
187 : "Re(1,1)", "Im(1,1)", "Re(2,-2)", "Im(2,-2)", "Re(2,-1)", "Im(2,-1)", "Re(2,0)",
188 : "Im(2,0)", "Re(2,1)", "Im(2,1)", "Re(2,2)", "Im(2,2)", ...
189 : ```
190 :
191 : Each dataset in the H5 file must also have an attribute
192 : named `Legend` which is an ASCII-encoded null-terminated variable-length string.
193 :
194 : #### Spherical harmonic nodes {#spherical_nodes}
195 :
196 : When we refer to a "nodal" data format, we mean that the worldtube data are
197 : stored as values at specially chosen collocation points (a.k.a. grid points or
198 : nodes). This allows SpECTRE to perform integrals, derivatives, and interpolation
199 : exactly on the input data. These grid points are Gauss-Legendre in $cos(\theta)$
200 : and equally spaced in $\phi$.
201 :
202 : Below is a routine for computing the spherical
203 : harmonic $\theta$ and $\phi$ values. These can be used to compute the Cartesian
204 : locations for a given radius using the standard transformation. The routine
205 : supports \f$\ell\in[4, 32]\f$.
206 :
207 : <details id="details">
208 : <summary> C Code for computing SpECTRE CCE gridpoint locations </summary>
209 : \snippet Test_Spherepack.cpp spectre_cce_grid_point_locations
210 : </details>
211 :
212 : Alternatively, if your code can read in grid points from a text file, you can
213 : run the `WriteCceWorldtubeCoordsToFile` executable like so to get a text file
214 : with three columns for the x,y,z coordinates of each point.
215 :
216 : ```
217 : ./WriteCceWorldtubeCoordsToFile -r 200 -L 16 -o GridPointsR200.txt
218 : ```
219 :
220 : Each dataset holds `1 + (l_max + 1) * (2 * l_max + 1)` columns, with the
221 : first one being the `time`. The columns must be in \f$\theta\f$-varies-fastest
222 : ordering. That is,
223 :
224 : ```
225 : "time",
226 : "Phi_0_Theta_0", "Phi_0_Theta_1", "Phi_0_Theta_2", "Phi_0_Theta_3", "Phi_0_Theta_4",
227 : "Phi_1_Theta_0", "Phi_1_Theta_1", "Phi_1_Theta_2", "Phi_1_Theta_3", "Phi_1_Theta_4",
228 : ```
229 :
230 : Each dataset in the H5 file must also have an attribute
231 : named `Legend` which is an ASCII-encoded null-terminated variable-length string.
232 :
233 : \note Nodal data is likely the easiest to write out since no conversion to
234 : spherical harmonic coefficients is necessary.
235 :
236 : #### ADM Cartesian metric and derivatives {#adm_cartesian_metric_and_derivatives}
237 :
238 : For worldtube data stored in an H5 file in the "ADM metric nodal" format, there
239 : must be the following datasets with these exact names (including the `.dat`
240 : suffix):
241 :
242 : - `gxx.dat`, `gxy.dat`, `gxz.dat`, `gyy.dat`, `gyz.dat`, `gzz.dat`
243 : - `Dxgxx.dat`, `Dxgxy.dat`, `Dxgxz.dat`, `Dxgyy.dat`, `Dxgyz.dat`, `Dxgzz.dat`
244 : - `Dygxx.dat`, `Dygxy.dat`, `Dygxz.dat`, `Dygyy.dat`, `Dygyz.dat`, `Dygzz.dat`
245 : - `Dzgxx.dat`, `Dzgxy.dat`, `Dzgxz.dat`, `Dzgyy.dat`, `Dzgyz.dat`, `Dzgzz.dat`
246 : - `Shiftx.dat`, `Shifty.dat`, `Shiftz.dat`
247 : - `DxShiftx.dat`, `DxShifty.dat`, `DxShiftz.dat`
248 : - `DyShiftx.dat`, `DyShifty.dat`, `DyShiftz.dat`
249 : - `DzShiftx.dat`, `DzShifty.dat`, `DzShiftz.dat`
250 : - `Lapse.dat`, `DxLapse.dat`, `DyLapse.dat`, `DzLapse.dat`
251 : - `Kxx.dat`, `Kxy.dat`, `Kxz.dat`, `Kyy.dat`, `Kyz.dat`, `Kzz.dat`
252 : - Either: `AuxiliaryShiftx.dat`, `AuxiliaryShifty.dat`, `AuxiliaryShiftz.dat`
253 : - Or: `ConformalChristoffelx.dat`, `ConformalChristoffely.dat`,
254 : `ConformalChristoffelz.dat`
255 :
256 : Here `g` represents the spacetime metric, but we only require the spatial
257 : components (e.g. `gxx.dat`, `gxy.dat`, etc...) so in practice, those are the
258 : tensor components of the spatial metric. The temporal components of the
259 : spacetime metric are stored separately in the lapse and shift. Each of the
260 : spatial metric, lapse, and shift must also have their cartesian derivatives. `K`
261 : is the extrinsic curvature, `AuxiliaryShift` is the auxiliary shift vector used
262 : in the first-order form of the Gamma-driver condition, and
263 : `ConformalChristoffel` is the trace of the conformal second_order symbols.
264 :
265 : We will compute the time derivative of the spatial metric using Eq. (2.134) of
266 : \cite BaumgarteShapiro,
267 :
268 : \begin{equation}
269 : \partial_t \gamma_{ij} = -2\alpha K_{ij} + D_i\beta_j + D_j\beta_i.
270 : \end{equation}
271 :
272 : The time derivative of the lapse is computed using the `1+log` slicing condition
273 : from Eq. (4.87) of \cite BaumgarteShapiro
274 :
275 : \begin{equation}
276 : \partial_t \alpha = -2\alpha K + \beta^j\partial_j\alpha,
277 : \end{equation}
278 :
279 : and the time derivative of the shift is computed using either the first order
280 : reduction form of the Gamma-driver condition from Eq. (4.89) of
281 : \cite BaumgarteShapiro
282 :
283 : \begin{equation}
284 : \partial_t \beta^i = \eta B^i + \beta^j\partial_j\beta^i,
285 : \end{equation}
286 :
287 : where you can choose $\eta$ (typically $\eta=0.75$) or using the integrated
288 : Gamma-driver condition from Eq. (12) of \cite Hilditch:2012fp
289 :
290 : \begin{equation}
291 : \partial_t \beta^i = \tilde{\Gamma}^i - \eta\beta^i + \beta^j\partial_j\beta^i,
292 : \end{equation}
293 :
294 : again, where you can choose $\eta$ (typically $\eta=2/M_{\textrm{ADM}}$).
295 :
296 : \warning If your worldtube data is in the ADM metric nodal format but you have
297 : not used `1+log` slicing and the Gamma-driver conditions specified above, your
298 : time derivatives will be **wrong**. If you'd like us to support other commonly
299 : used gauge conditions (or variants of `1+log` or Gamma-driver), please open an
300 : issue on our
301 : [GitHub](https://github.com/sxs-collaboration/spectre).
302 :
303 : The layout of each of these datasets must be
304 : [spherical harmonic nodes](#spherical_nodes).
305 :
306 : #### Cartesian metric and derivatives {#cartesian_metric_and_derivatives}
307 :
308 : For worldtube data stored in an H5 file in either the "metric nodal" or "metric
309 : modal" formats, there must be the following datasets with these exact names
310 : (including the `.dat` suffix):
311 :
312 : - `gxx.dat`, `gxy.dat`, `gxz.dat`, `gyy.dat`, `gyz.dat`, `gzz.dat`
313 : - `Drgxx.dat`, `Drgxy.dat`, `Drgxz.dat`, `Drgyy.dat`, `Drgyz.dat`, `Drgzz.dat`
314 : - `Dtgxx.dat`, `Dtgxy.dat`, `Dtgxz.dat`, `Dtgyy.dat`, `Dtgyz.dat`, `Dtgzz.dat`
315 : - `Shiftx.dat`, `Shifty.dat`, `Shiftz.dat`
316 : - `DrShiftx.dat`, `DrShifty.dat`, `DrShiftz.dat`
317 : - `DtShiftx.dat`, `DtShifty.dat`, `DtShiftz.dat`
318 : - `Lapse.dat`
319 : - `DrLapse.dat`
320 : - `DtLapse.dat`
321 :
322 : Here `g` represents the spacetime metric, but we only require the spatial
323 : components (e.g. `gxx.dat`, `gxy.dat`, etc...) so in practice, those are the
324 : tensor components of the spatial metric. The temporal components of the
325 : spacetime metric are stored separately in the lapse and shift. The layout of
326 : each of these datasets must be in either
327 : [spherical harmonic modes](#spherical_modes) or
328 : [spherical harmonic nodes](#spherical_nodes).
329 :
330 : #### Bondi-Sachs {#bondi_sachs}
331 :
332 : In the "bondi nodal" format, you must have the same Bondi variables as the
333 : [required format](#required_h5_worldtube_data_format), but each variable layout
334 : must be the [spherical harmonic nodal layout](#spherical_nodes) with complex
335 : values interleaved as `Re`, `Im`, `Re`, `Im`, ...
336 :
337 : If you already have data in the
338 : [required "bondi modal" format](#required_h5_worldtube_data_format), then
339 : nothing needs to be done.
340 :
341 : #### Running the PreprocessCceWorldtube executable
342 :
343 : The `PreprocessCceWorldtube` executable should be run on any of the
344 : [allowed input formats](#converting_worldtube_data), and will produce a
345 : corresponding Bondi-Sachs worldtube file that can be read in by CCE. This
346 : executable works similarly to our other executables by accepting a YAML input
347 : file:
348 :
349 : ```
350 : PreprocessCceWorldtube --input-file PreprocessCceWorldtube.yaml
351 : ```
352 :
353 : with a YAML file
354 :
355 : \snippet PreprocessCceWorldtube.yaml preprocess_cce_worldtube_yaml_doxygen_example
356 :
357 : In addition to converting worldtube data formats, `PreprocessCceWorldtube` also
358 : accepts multiple input worldtube H5 files that have sequential times (e.g. from
359 : different checkpoints) and will combine the times from all H5 files alongside
360 : converting the worldtube data format. If there are duplicate or overlapping
361 : times, the last/latest of the times are chosen. If you pass multiple input
362 : worldtube H5 files, it is assumed that they are ordered increasing in time.
363 :
364 : Here are some notes about the different options in the YAML input file:
365 :
366 : - If the extraction radius is in the `InputH5File` names, then the
367 : `ExtractionRadius` option can be `Auto`. Otherwise, it must be specified.
368 : - The option `LMaxFactor` determines the factor by which the resolution of the
369 : boundary computation that is run will exceed the resolution of the input and
370 : output files. Empirically, we have found that `LMaxFactor` of 3 is sufficient
371 : to achieve roundoff precision in all boundary data we have attempted, and an
372 : `LMaxFactor` of 2 is usually sufficient to vastly exceed the precision of the
373 : simulation that provided the boundary dataset.
374 : - `FixSpecNormalization` should always be `False` unless you are using a
375 : particualy old version of SpEC
376 : - `DescendingM` should always be `False`. This option is to support metric modal
377 : data from SpEC. Our [data format](#input_worldtube_data_format) is always in
378 : *ascending* m.
379 : - `BufferDepth` is an advanced option that lets you load more data into RAM at
380 : once so there are fewer filesystem accesses.
381 : - If using the ADM metric nodal input data format with a first-order form gamma
382 : driver, specify the `InputDataFormat:` like so:
383 : \snippet AdmFirstOrderDriverPreprocessCceWorldtube.yaml first_order_input_data_format_example
384 : - If using the ADM metric nodal input data format with an integrated gamma
385 : driver, specify the `InputDataFormat:` like so:
386 : \snippet AdmSecondOrderDriverPreprocessCceWorldtube.yaml second_order_input_data_format_example
387 :
388 : ### What Worldtube data "should" look like {#worldtube_data_looks}
389 :
390 : While no two simulations will look exactly the same, there are some general
391 : trends in the worldtube data to look for. Here is a plot of some modes of the
392 : Bondi variable `J` from the [Bondi-Sachs](#bondi_sachs) worldtube format.
393 :
394 : \image html worldtube_J.png "Bondi variable J on the Worldtube"
395 :
396 : The 2,2 modes are oscillatory and capture the orbits of the two objects. The
397 : real part of the 2,0 mode contains the gravitational memory of the system. Then
398 : for this system, all the other modes are subdominant.
399 :
400 : If you are using the [cartesian metric](#cartesian_metric_and_derivatives) or
401 : [adm cartesian metric](#adm_cartesian_metric_and_derivatives)
402 : worldtube format, here is a plot of the imaginary part of the 2,2 mode of the
403 : lapse and its radial and time derivative during inspiral.
404 :
405 : \image html lapse.png "2,2 component of lapse and its radial and time derivative"
406 :
407 : One thing to keep in mind with this plot is that it was produced using the
408 : Generalized Harmonic formulation of Einstein's equations using the damped
409 : harmonic gauge. Therefore, if you are using a different formulation and gauge
410 : (like BSSN + moving punctures), the lapse may look different than this. One way
411 : to sanity check your data (regardless of what type it is or where you got it
412 : from) is to look and how the mode amplitude for a given quantity decays as you
413 : increase (l,m). Here is a plot of the amplitude of the modes for the lapse in
414 : the above plot.
415 :
416 : \image html amp_lapse.png "Amplitude of modes of Lapse"
417 :
418 : You'll notice that most modes are around machine precision and only the first
419 : few have any real impact. This is expected.
420 :
421 : ## Input file for CCE
422 :
423 : Input files for CCE are commonly named `CharacteristicExtract.yaml`. An example
424 : input file with comments explaining some of the options can be found in
425 : `$SPECTRE_HOME/tests/InputFiles/Cce/CharacteristicExtract.yaml`. Here we expand
426 : a bit on why we chose some of those parameters.
427 :
428 : ### General options
429 :
430 : - For resolution, the example input file has lmax (`Cce.LMax`) of 20, and
431 : filter lmax (`Filtering.FilterLMax`) of 18; that may run a bit slow for
432 : basic tests, but this value yields the best precision-to-run-time ratio
433 : for a typical BBH system. Note that precision doesn't improve above lmax 24,
434 : filter 22 (be sure to update the filter as you update lmax -- the filter
435 : should generally be around 2 lower than the maximum l to reduce possible
436 : aliasing).
437 : - If you want to just run over all times in the worldtube H5 file, you can
438 : set both the `StartTime` and `EndTime` to `Auto` and it will automatically
439 : figure it out based on the data in the worldtube file.
440 : - The `ScriOutputDensity` adds extra interpolation points to the output,
441 : which is useful for finite-difference derivatives on the output data, but
442 : otherwise it'll just unnecessarily inflate the output files, so if you
443 : don't need the extra points, best just set it to 1.
444 : - For production level runs, it's recommended to have the
445 : `Cce.Evolution.StepChoosers.Constant` option set to 0.1 for an accurate time
446 : evolution. However, if you're just testing, this can be increased to 0.5 to
447 : speed things up.
448 : - We generally do not recommend extracting at less than 100M due to the CCE junk
449 : radiation being much worse at these smaller worldtube radii. That being said,
450 : we also recommend running CCE over several worldtube radii and checking which
451 : is the best based on the Bianchi identity violations. There isn't necessarily
452 : a "best radius" to extract waveforms at.
453 : - Most users will not need it, but if you want to dump data from the volume
454 : (instead of only on future null infinity), this is possible with the
455 : `ObserveFields` option to `Events` within `EventsAndTriggersAtSlabs` in the
456 : input file. See the input file referenced above for a (commented-out) example,
457 : and the documentation of the class `Cce::Events::ObserveFields` for details.
458 :
459 : ### Initial data on the null hypersurface
460 :
461 : Choosing initial data on the initial null hypersurface is a non-trivial task and
462 : is an active area of research. We want initial data that will reduce the amount
463 : of CCE junk radiation as much as possible, while also having the initial data
464 : work for as many cases as possible.
465 :
466 : SpECTRE currently has four different methods to choose the initial data on
467 : the null hypersurface. In order from most recommended to least recommended,
468 : these are:
469 :
470 : - `ConformalFactor`: Try to make initial time coordinate as inertial as
471 : possible at \f$\mathscr{I}^+\f$ with a smart choice of the conformal factor.
472 : This will work for many cases, but not all. But will produce the best initial
473 : data when it does work.
474 : - `InverseCubic`: Ansatz where \f$J = A/r + B/r^3\f$. This is very robust and
475 : almost never fails, but contains a lot of CCE junk radiation compared to
476 : `ConformalFactor`.
477 : - `ZeroNonSmooth`: Make `J` vanish. Like the name says, it's not smooth.
478 : - `NoIncomingRadiation`: Make \f$\Psi_0 = 0\f$; this does not actually lead
479 : to no incoming radiation, since \f$\Psi_0\f$ and \f$\Psi_4\f$ both include
480 : incoming and outgoing radiation.
481 :
482 : ### Rechunking worldtube data
483 :
484 : \note This section is less important than the others and really only matters if
485 : you will be doing a large number of CCE runs like for a catalog. For only a
486 : couple runs or just for testing, this part is unnecessary and can be skipped.
487 :
488 : CCE will run faster if the input worldtube hdf5 file is chunked in small numbers
489 : of complete rows. This is relevant because by default, SpEC and SpECTRE write
490 : their worldtube files chunked along full time-series columns, which is
491 : efficient for writing and compression, but not for reading in to CCE. In that
492 : case, you can rechunk the input file before running CCE for maximum performance.
493 : This can be done, for instance, using h5py (you will need to fill in filenames
494 : appropriate to your case in place of "BondiCceR0050.h5" and
495 : "RechunkBondiCceR0050.h5"):
496 :
497 : ```py
498 : import h5py
499 : input_file = "BondiCceR0050.h5"
500 : output_file = "RechunkBondiCceR0050.h5"
501 : with h5py.File(input_file,'r') as input_h5,\
502 : h5py.File(output_file, 'w') as output_h5:
503 : for dset in input_h5:
504 : if("Version" in dset):
505 : output_h5[dset] = input_h5[dset][()]
506 : continue
507 : number_of_columns = input_h5[dset][()].shape[1]
508 : output_h5.create_dataset(dset, data=input_h5[dset],
509 : maxshape=(None, number_of_columns),
510 : chunks=(4, number_of_columns), dtype='d')
511 : for attribute in input_h5[dset].attrs.keys():
512 : output_h5[dset].attrs[attribute] = input_h5[dset].attrs[attribute]
513 : ```
514 :
515 : The rechunked data will still be in the same
516 : [format](#input_worldtube_data_format) as before, but will just have a
517 : different underlying structure in the H5 file that makes it faster to read in.
518 :
519 : ## Running the CCE executable {#running_the_cce_executable}
520 :
521 : Once you have [acquired an executable](#acquiring_the_cce_module), running CCE
522 : in a supported environment is a simple command:
523 :
524 : ```
525 : ./CharacteristicExtract --input-file CharacteristicExtract.yaml
526 : ```
527 :
528 : You may notice at the beginning you get some warnings that look like
529 :
530 : ```
531 : Warning: iterative angular solve did not reach target tolerance 1.000000e-13.
532 : Exited after 300 iterations, achieving final maximum over collocation points
533 : for deviation from target of 2.073455e-08
534 : Proceeding with evolution using the partial result from partial angular solve.
535 : ```
536 :
537 : This is normal and expected. All it means is that initially an angular solve
538 : didn't hit a tolerance. We've found that it never really reaches the tolerance
539 : of 1e-13, but we still keep this tolerance so it gets as low as possible.
540 :
541 : After this, you'll likely see some output like
542 :
543 : ```
544 : Simulation time: 10.000000
545 : Wall time: 00:00:44
546 : Simulation time: 20.000000
547 : Wall time: 00:01:14
548 : ```
549 :
550 : which tells you that the simulation is proceeding as expected. When the run
551 : finished, you'll see something like
552 :
553 : ```
554 : Done!
555 : Wall time: 06:01:41
556 : Date and time at completion: Thu May 23 22:31:27 2024
557 :
558 : [Partition 0][Node 0] End of program
559 : ```
560 :
561 : In terms of runtime, we've found that for a ~5000M long cauchy simulation, CCE
562 : takes about 6 hours to run. This will vary based on a number of factors like how
563 : long the cauchy evolution actually is, the desired error tolerance of your
564 : characteristic timestepper, and also how much data you output at future null
565 : infinity. Therefore, take these numbers with a grain of salt and only use them
566 : as a rough estimate for how long a job will take.
567 :
568 : \note CCE can technically run on two (2) cores by adding the option `++ppn 2` to
569 : the above command, however, we have found in practice that this makes
570 : little-to-no difference in the runtime of the executable.
571 :
572 : ## Output from CCE
573 :
574 : Once you have the reduction data output file from a successful CCE run, you can
575 : confirm the integrity of the h5 file and its contents by running
576 :
577 : ```
578 : h5ls -r CharacteristicExtractReduction.h5
579 : ```
580 :
581 : For the reduction file produced by a successful run, the output of the `h5ls`
582 : should resemble
583 :
584 : ```
585 : /SpectreR0100.cce Group
586 : /SpectreR0100.cce/EthInertialRetardedTime Dataset {26451/Inf, 163}
587 : /SpectreR0100.cce/News Dataset {26451/Inf, 163}
588 : /SpectreR0100.cce/Psi0 Dataset {26451/Inf, 163}
589 : /SpectreR0100.cce/Psi1 Dataset {26451/Inf, 163}
590 : /SpectreR0100.cce/Psi2 Dataset {26451/Inf, 163}
591 : /SpectreR0100.cce/Psi3 Dataset {26451/Inf, 163}
592 : /SpectreR0100.cce/Psi4 Dataset {26451/Inf, 163}
593 : /SpectreR0100.cce/Strain Dataset {26451/Inf, 163}
594 : /src.tar.gz Dataset {7757329}
595 : ```
596 :
597 : Notice that the worldtube radius will be encoded into the subfile name.
598 :
599 : \note Prior to
600 : [this Pull Request](https://github.com/sxs-collaboration/spectre/pull/5985),
601 : merged May 15, 2024, the output of `h5ls` looked like this
602 : ```
603 : / Group
604 : /Cce Group
605 : /Cce/EthInertialRetardedTime.dat Dataset {3995/Inf, 163}
606 : /Cce/News.dat Dataset {3995/Inf, 163}
607 : /Cce/Psi0.dat Dataset {3995/Inf, 163}
608 : /Cce/Psi1.dat Dataset {3995/Inf, 163}
609 : /Cce/Psi2.dat Dataset {3995/Inf, 163}
610 : /Cce/Psi3.dat Dataset {3995/Inf, 163}
611 : /Cce/Psi4.dat Dataset {3995/Inf, 163}
612 : /Cce/Strain.dat Dataset {3995/Inf, 163}
613 : /src.tar.gz Dataset {3750199}
614 : ```
615 :
616 : No matter the spin weight of the quantity, the $\ell = 0,1$ modes will always be
617 : output by CCE. Also, the `/SpectreR0100.cce` group will have a legend attribute
618 : that specifies the columns.
619 :
620 : ### Raw CCE output
621 :
622 : The `Strain` represents the asymptotic transverse-traceless contribution
623 : to the metric scaled by the Bondi radius (to give the asymptotically leading
624 : part), the `News` represents the first time derivative of the strain, and each
625 : of the `Psi...` datasets represent the Weyl scalars, each scaled by the
626 : appropriate factor of the Bondi-Sachs radius to retrieve the asymptotically
627 : leading contribution.
628 :
629 : The `EthInertialRetardedTime` is a diagnostic dataset that represents the
630 : angular derivative of the inertial retarded time, which determines the
631 : coordinate transformation that is performed at future null infinity.
632 :
633 : If you'd like to visualize the output of a CCE run, we offer a
634 : [CLI](py/cli.html) that will produce a plot of all of quantities except
635 : `EthInertialRetardedTime`. To see how to use this CLI, run
636 :
637 : ```
638 : spectre plot cce -h
639 : ```
640 :
641 : If you'd like to do something more complicated than just make a quick plot,
642 : you'll have to load in the output data yourself using `h5py` or our
643 : `spectre.IO.H5` bindings.
644 :
645 : \note The CLI can also plot the "old" version of CCE output, described above.
646 : Pass `--cce-group %Cce` to the CLI. This option is only for backwards
647 : compatibility with the old CCE output and is not supported for the current
648 : version of output. This options is deprecated and will be removed in the future.
649 :
650 : ### Frame fixing
651 :
652 : You may notice some odd features in some of the output quantities if you try and
653 : plot them. This is not a bug in CCE (not that we know of at least). This occurs
654 : because the data at future null infinity is in the wrong Bondi-Metzner-Sachs
655 : (BMS) frame. In order to put this data into the correct BMS frame, the
656 : [SXS Collaboration](https://github.com/sxs-collaboration) offers a python/numba
657 : code called [scri](https://github.com/moble/scri) to do these transformations.
658 : See their documentation for how to install/run/plot a waveform at future null
659 : infinity in the correct BMS frame.
660 :
661 : Below is a plot of the imaginary part of the 2,2 component for the strain when
662 : plotted using the raw output from SpECTRE CCE and BMS frame-fixing with scri.
663 : You'll notice that there is a non-zero offset, and this component of the strain
664 : doesn't decay back down to zero during the ringdown. This is because of the
665 : improper BMS frame that the SpECTRE CCE waveform is in. A supertranslation must
666 : be applied to transform the waveform into the correct BMS frame. See
667 : \cite Mitman2024review for a review of BMS transformations and gravitational
668 : memory. To perform the frame fixing, see
669 : [this tutorial](https://scri.readthedocs.io/en/latest/tutorial_abd.html#loading-cce-data-and-adjusting-the-bms-frame)
670 : in scri.
671 :
672 : \image html im_h22.png "Imaginary part of 2,2 component of the strain"
673 :
674 : The discrepancy is even more apparent if you plot the amplitude of the two
675 : waveforms. It's pretty clear which waveform is the more "physical" one.
676 :
677 : \image html amp_im_h22.png "Amplitude of imaginary part of 2,2 component of the strain"
678 :
679 : Notice that there are still some oscillations in the strain output by scri
680 : towards the beginning of the waveform (up to ~1500M). This is caused by CCE junk
681 : radiation from imperfect initial data on the null hypersurface. In order to use
682 : this waveform in analysis, the CCE junk must be cut off from the beginning.
683 :
684 : You are also able to see gravitational memory effects with SpECTRE CCE! This
685 : shows up in the real part of the 2,0 mode of the strain. Though you can see the
686 : memory effects in the SpECTRE CCE waveform, in order to do any analysis, you
687 : must also transform the waveform to a more physically motivated BMS frame.
688 :
689 : \image html re_h20.png "Real part of 2,0 component of the strain
690 :
691 : ## Citing CCE
692 :
693 : If you use the SpECTRE CCE module to extract your waveforms at future null
694 : infinity, please cite the following:
695 :
696 : - [SpECTRE DOI](https://zenodo.org/doi/10.5281/zenodo.4290404) (This link
697 : defaults to the latest release of SpECTRE. From there, you can find links to
698 : past releases of SpECTRE as well)
699 : - [SpECTRE CCE paper](https://doi.org/10.1103/PhysRevD.107.064013)
700 : - [CCE paper](https://doi.org/10.1103/PhysRevD.102.044052)
701 :
702 : If you used scri to perform any frame-fixing, please also consult the
703 : [scri GitHub](https://github.com/moble/scri) for how you should cite it.
704 :
705 : You can also consult our \ref publication_policies page for further questions or
706 : contact [spectre-devel@black-holes.org](mailto:spectre-devel@black-holes.org).
|
