SpECTRE
v2024.09.29
|
There are a couple different ways to acquire the CCE module/executable.
Starting from late May 2024, in every Release of SpECTRE we offer a tarball that contains everything needed to run CCE on a large number of different systems. This will be under the Assets
section towards the bottom of the release (there may be a lot of text detailing what's been updated in this release). Inside this tarball is
CharacteristicExtract
Tests/
directory (see Input worldtube data formats section)Tests/
directoryReduceCceWorldtube
executable and YAML file for converting between worldtube data formats in the ReduceCceWorldtube/
diretoryCheckCceOutput.py
(meant to be run from the root of the tarball and after you run the example YAML input file also in the root of the tarball) that will check if the example output is correct.xz
so use tar -xf TarName.tar.xz
to extract. The -z
flag to use gzip will cause an error.See Running the CCE executable for how to run CCE.
We have tested that this executable works natively on the following machines:
LD_LIBRARY_PATH=/sw/spack/deltas11-2023-03/apps/linux-rhel8-x86_64/gcc-8.5.0/gcc-11.4.0-yycklku/lib64/:$LD_LIBRARY_PATH
before running CCE)We have also tested that this executable works inside our dev
Docker container on the following machines (in addition to the ones above):
You can clone the spectre repo and follow the instructions on the Installation page to obtain an environment to configure and build SpECTRE. Once you have a configured build
directory, build the CCE executable with
-j4
flag to speed up compilation. However, be warned that this executable will need several GB of memory to build.The worldtube data must be constructed as spheres of constant coordinate radius, and (for the time being) written to a filename of the format ...CceRXXXX.h5
, where the XXXX
is to be replaced by the integer for which the extraction radius is equal to XXXX
M. For instance, a 100M extraction should have filename ...CceR0100.h5
. This scheme of labeling files with the extraction radius is constructed for compatibility with SpEC worldtube data.
Currently CCE is able to read in worldtube data in two different formats.
This metric data format must be provided as the following datasets:
gxx.dat
, gxy.dat
, gxz.dat
, gyy.dat
, gyz.dat
, gzz.dat
Drgxx.dat
, Drgxy.dat
, Drgxz.dat
, Drgyy.dat
, Drgyz.dat
, Drgzz.dat
Dtgxx.dat
, Dtgxy.dat
, Dtgxz.dat
, Dtgyy.dat
, Dtgyz.dat
, Dtgzz.dat
Shiftx.dat
, Shifty.dat
, Shiftz.dat
DrShiftx.dat
, DrShifty.dat
, DrShiftz.dat
DtShiftx.dat
, DtShifty.dat
, DtShiftz.dat
Lapse.dat
DrLapse.dat
DtLapse.dat
In this format, the worldtube data are stored as spherical harmonic coefficients. We use spherical harmonic conventions documented by the ylm::Spherepack class. Each row must start with the time stamp, and the remaining values are the complex modes in m-varies-fastest format. That is,
In this format the value of the functions at specially chosen collocation points (grid points) is read in. This allows SpECTRE to perform integrals, derivatives, and interpolation exactly on the input data. These grid points are Gauss-Legendre in
Each dat
file holds 1 + (l_max + 1) * (2 * l_max + 1)
columns, with the first one being the time
. The columns must be in
Each dataset in the file must also have an attribute named "Legend" which is an ASCII-encoded null-terminated variable-length string. That is, the HDF5 type is:
This can be checked for a dataset by running
The second format is Bondi-Sachs metric component data. This format is far more space-efficient (by around a factor of 4) than the cartesian_metric format.
The format is similar to the cartesian_metric format, except in spin-weighted spherical harmonic modes, and the real (spin-weight-0) quantities omit the redundant negative-m modes and imaginary parts of m=0 modes. The quantities that must be provided by the Bondi-Sachs metric data format are:
Beta.dat
DrJ.dat
DuR.dat
H.dat
J.dat
Q.dat
R.dat
U.dat
W.dat
An example of the columns of these dat files is
See [141] for a description of these quantities.
Since the Bondi-Sachs format is far more space-efficient, SpECTRE provides a separate executable for converting from the cartesian_metric format to the Bondi-Sachs worldtube format called ReduceCceWorldtube
. The ReduceCceWorldtube
executable should be run on a cartesian_metric worldtube file, and will produce a corresponding 'reduced' Bondi-Sachs worldtube file. This executable works similarly to our other executables by accepting a YAML input file:
with a YAML file
The option LMaxFactor
determines the factor by which the resolution of the boundary computation that is run will exceed the resolution of the input and output files. Empirically, we have found that LMaxFactor
of 3 is sufficient to achieve roundoff precision in all boundary data we have attempted, and an LMaxFactor
of 2 is usually sufficient to vastly exceed the precision of the simulation that provided the boundary dataset.
While no two simulations will look exactly the same, there are some general trends in the worldtube data to look for. Here is a plot of some modes of the Bondi variable J
from the Bondi-Sachs worldtube format.
The 2,2 modes are oscillatory and capture the orbits of the two objects. The real part of the 2,0 mode contains the gravitational memory of the system. Then for this system, all the other modes are subdominant.
If you are using the cartesian metric worldtube format, here is a plot of the imaginary part of the 2,2 mode of the lapse and its radial and time derivative during inspiral.
One thing to keep in mind with this plot is that it was produced using the Generalized Harmonic formulation of Einstein's equations using the damped harmonic gauge. Therefore, if you are using a different formulation and gauge (like BSSN + moving punctures), the lapse may look different than this. One way to sanity check your data (regardless of what type it is or where you got it from) is to look and how the mode amplitude for a given quantity decays as you increase (l,m). Here is a plot of the amplitude of the modes for the lapse in the above plot.
You'll notice that most modes are around machine precision and only the first few have any real impact. This is expected.
Input files for CCE are commonly named CharacteristicExtract.yaml
. An example input file with comments explaining some of the options can be found in $SPECTRE_HOME/tests/InputFiles/Cce/CharacteristicExtract.yaml
. Here we expand a bit on why we chose some of those parameters.
Cce.LMax
) of 20, and filter lmax (Filtering.FilterLMax
) of 18; that may run a bit slow for basic tests, but this value yields the best precision-to-run-time ratio for a typical BBH system. Note that precision doesn't improve above lmax 24, filter 22 (be sure to update the filter as you update lmax – the filter should generally be around 2 lower than the maximum l to reduce possible aliasing).StartTime
and EndTime
to Auto
and it will automatically figure it out based on the data in the worldtube file.ScriOutputDensity
adds extra interpolation points to the output, which is useful for finite-difference derivatives on the output data, but otherwise it'll just unnecessarily inflate the output files, so if you don't need the extra points, best just set it to 1.Cce.Evolution.StepChoosers.Constant
option set to 0.1 for an accurate time evolution. However, if you're just testing, this can be increased to 0.5 to speed things up.Cce.H5IsBondiData
to True
. If the worldtube data is the cartesian_metric form, set Cce.H5IsBondiData
to False
.Choosing initial data on the initial null hypersurface is a non-trivial task and is an active area of research. We want initial data that will reduce the amount of CCE junk radiation as much as possible, while also having the initial data work for as many cases as possible.
SpECTRE currently has four different methods to choose the initial data on the null hypersurface. In order from most recommended to least recommended, these are:
ConformalFactor
: Try to make initial time coordinate as inertial as possible at InverseCubic
: Ansatz where ConformalFactor
.ZeroNonSmooth
: Make J
vanish. Like the name says, it's not smooth.NoIncomingRadiation
: Make CCE will run faster if the input worldtube hdf5 file is chunked in small numbers of complete rows. This is relevant because by default, SpEC and SpECTRE write their worldtube files chunked along full time-series columns, which is efficient for writing and compression, but not for reading in to CCE. In that case, you can rechunk the input file before running CCE for maximum performance. This can be done, for instance, using h5py (you will need to fill in filenames appropriate to your case in place of "BondiCceR0050.h5" and "RechunkBondiCceR0050.h5"):
The rechunked data will still be in the same format as before, but will just have a different underlying structure in the H5 file that makes it faster to read in.
Once you have acquired an executable, running CCE in a supported environment is a simple command:
You may notice at the beginning you get some warnings that look like
This is normal and expected. All it means is that initially an angular solve didn't hit a tolerance. We've found that it never really reaches the tolerance of 1e-13, but we still keep this tolerance so it gets as low as possible.
After this, you'll likely see some output like
which tells you that the simulation is proceeding as expected. When the run finished, you'll see something like
In terms of runtime, we've found that for a ~5000M long cauchy simulation, CCE takes about 6 hours to run. This will vary based on a number of factors like how long the cauchy evolution actually is, the desired error tolerance of your characteristic timestepper, and also how much data you output at future null infinity. Therefore, take these numbers with a grain of salt and only use them as a rough estimate for how long a job will take.
++ppn 2
to the above command, however, we have found in practice that this makes little-to-no difference in the runtime of the executable.Once you have the reduction data output file from a successful CCE run, you can confirm the integrity of the h5 file and its contents by running
For the reduction file produced by a successful run, the output of the h5ls
should resemble
Notice that the worldtube radius will be encoded into the subfile name.
h5ls
looked like this No matter the spin weight of the quantity, the /SpectreR0100.cce
group will have a legend attribute that specifies the columns.
The Strain
represents the asymptotic transverse-traceless contribution to the metric scaled by the Bondi radius (to give the asymptotically leading part), the News
represents the first time derivative of the strain, and each of the Psi...
datasets represent the Weyl scalars, each scaled by the appropriate factor of the Bondi-Sachs radius to retrieve the asymptotically leading contribution.
The EthInertialRetardedTime
is a diagnostic dataset that represents the angular derivative of the inertial retarded time, which determines the coordinate transformation that is performed at future null infinity.
If you'd like to visualize the output of a CCE run, we offer a CLI that will produce a plot of all of quantities except EthInertialRetardedTime
. To see how to use this CLI, run
If you'd like to do something more complicated than just make a quick plot, you'll have to load in the output data yourself using h5py
or our spectre.IO.H5
bindings.
--cce-group Cce
to the CLI. This option is only for backwards compatibility with the old CCE output and is not supported for the current version of output. This options is deprecated and will be removed in the future.You may notice some odd features in some of the output quantities if you try and plot them. This is not a bug in CCE (not that we know of at least). This occurs because the data at future null infinity is in the wrong Bondi-Metzner-Sachs (BMS) frame. In order to put this data into the correct BMS frame, the SXS Collaboration offers a python/numba code called scri to do these transformations. See their documentation for how to install/run/plot a waveform at future null infinity in the correct BMS frame.
Below is a plot of the imaginary part of the 2,2 component for the strain when plotted using the raw output from SpECTRE CCE and BMS frame-fixing with scri. You'll notice that there is a non-zero offset, and this component of the strain doesn't decay back down to zero during the ringdown. This is because of the improper BMS frame that the SpECTRE CCE waveform is in. A supertranslation must be applied to transform the waveform into the correct BMS frame. See [137] for a review of BMS transformations and gravitational memory. To perform the frame fixing, see this tutorial in scri.
The discrepancy is even more apparent if you plot the amplitude of the two waveforms. It's pretty clear which waveform is the more "physical" one.
Notice that there are still some oscillations in the strain output by scri towards the beginning of the waveform (up to ~1500M). This is caused by CCE junk radiation from imperfect initial data on the null hypersurface. In order to use this waveform in analysis, the CCE junk must be cut off from the beginning.
You are also able to see gravitational memory effects with SpECTRE CCE! This shows up in the real part of the 2,0 mode of the strain. Though you can see the memory effects in the SpECTRE CCE waveform, in order to do any analysis, you must also transform the waveform to a more physically motivated BMS frame.
If you use the SpECTRE CCE module to extract your waveforms at future null infinity, please cite the following:
If you used scri to perform any frame-fixing, please also consult the scri GitHub for how you should cite it.
You can also consult our Publication policies page for further questions or contact spectre-devel@black-holes.org.