Line data Source code
1 0 : \cond NEVER
2 : Distributed under the MIT License.
3 : See LICENSE.txt for details.
4 : \endcond
5 : # A Hitchhiker's Guide to Running SpECTRE {#beginners_guide}
6 :
7 : \tableofcontents
8 :
9 : SpECTRE can be a bit complicated to get started with, especially if you aren't
10 : familiar with our core concepts of task-based parallelism and Template
11 : Meta-Programming (TMP). However, <a
12 : href="https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Don't_Panic">
13 : Don't Panic</a>. This guide aims to get you introduced to running,
14 : visualizing, editing, and then rebuilding SpECTRE to give you a feel for what
15 : SpECTRE is all about, all on your own laptop! Hopefully by the end of this guide
16 : you'll feel comfortable enough to look at other executables and maybe even
17 : venture into the code itself!
18 :
19 : ## Prerequisites
20 :
21 : To start off, you'll need to obtain an environment to build and run SpECTRE in.
22 : You could try and install all the dependencies yourself, but that is very
23 : tedious and very error prone. Instead, we provide a
24 : [Docker](https://docs.docker.com/get-docker/) container with all the
25 : dependencies pre-installed for you to use. The container also has the SpECTRE
26 : repository cloned in it already so you don't have to worry about getting it
27 : yourself. To obtain the docker image, run
28 :
29 : ```
30 : docker pull sxscollaboration/spectre:demo
31 : ```
32 :
33 : Another program you will need for this tutorial is
34 : [Paraview](https://www.paraview.org/download/) for visualizing the output. You
35 : specifically will need version 5.10.1 for this tutorial.
36 :
37 : If you'd like to use VSCode, the tutorial also has instructions for how to start
38 : in VSCode as well.
39 :
40 : ## Into the Container
41 :
42 : For both a terminal and VSCode, create the container in a terminal and start it.
43 :
44 : ```
45 : docker create --entrypoint "/bin/bash" --rm --name spectre_demo -p 11111:11111 \
46 : -i -t sxscollaboration/spectre:demo
47 : ```
48 : ```
49 : docker start spectre_demo
50 : ```
51 :
52 : We connect port `11111` on your local machine to port `11111` of the container
53 : so we can use Paraview. The `--rm` will delete the container when you stop it.
54 : This won't put you into the container, only start it in the background.
55 :
56 : \note The `--entrypoint "/bin/bash"` is important because the default entrypoint
57 : of the container is the SpECTRE CLI. You can try out the default entrypoint by
58 : running `docker run sxscollaboration/spectre:demo -h` and it'll print the help
59 : string for the CLI. See the [Python documentation](py/cli.html) for more.
60 :
61 : You can also run a [Jupyter](https://jupyter.org/index.html) server for
62 : accessing the Python bindings (see \ref spectre_using_python) or running Jupyter
63 : notebooks. To do so, append another `-p` option with your specified port, e.g.
64 : `-p 8000:8000`. You can chain as many `-p` options as you want to expose more
65 : ports.
66 :
67 : The SpECTRE repository is located at `/work/spectre` inside the container.
68 :
69 :
70 : ### With a Terminal {#with_terminal}
71 :
72 : To hop in the container from a terminal, simply type
73 :
74 : ```
75 : docker attach spectre_demo
76 : ```
77 :
78 : and now you're in the container!
79 :
80 : ### With VSCode
81 :
82 : If you're using VSCode, you'll need the `Remote-Containers` extension to be
83 : able to access the container. Once you have it, open the
84 : [command palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette)
85 : and run the following commands.
86 :
87 : 1. `Remote-Containers: Attach to Running Container` - you should see the
88 : container `spectre_demo` that's currently running. Select that.
89 : 2. `File: Open Folder` - select `/work/spectre` which is where the repo is.
90 :
91 : Now you're in the container within VSCode! The terminal in VSCode will look
92 : identical to the one if you hadn't used VSCode.
93 :
94 : \note Any changes you make inside `/work/spectre` will be lost once you stop the
95 : container. If you'd like your changes to persist, get rid of the `--rm` flag in
96 : the `docker create` command.
97 :
98 : ## Compiling the code
99 :
100 : \note From here on out, all paths are assumed to be inside the container unless
101 : specified otherwise.
102 :
103 : The container already has a SpECTRE build pre-configured. Go to the build
104 : directory and compile the executables that we will use in this tutorial:
105 :
106 : ```sh
107 : cd /work/spectre/build
108 : make -j2 ExportCoordinates3D EvolveScalarAdvection2D all-pybindings
109 : ```
110 :
111 : This will compile the code on two cores. If you'd like to use more cores, use
112 : the `-j N` option where `N` is the number of cores.
113 :
114 : Once the executables are compiled they will be available in the
115 : `/work/spectre/build/bin` directory. The container already has this directory
116 : added to the `PATH` environment variable, so you can run executables from the
117 : command line right away:
118 :
119 : ```sh
120 : spectre --help
121 : ```
122 :
123 : If you are not in the container but instead on a desktop or cluster you can add
124 : the bin directory to your path by `cd`ing to the build directory and running
125 : ```
126 : export PATH=$PATH:`pwd`/bin
127 : ```
128 : If you log out and log back in you will need to set this again, unless you add
129 : it to your shell's init file.
130 :
131 : ## Running ExportCoordinates3D
132 :
133 : First we will run the `ExportCoordinates3D` executable to visualize
134 : the coordinates of a binary black hole domain.
135 : Make a directory where you will run everything:
136 :
137 : ```
138 : mkdir /work/runs
139 : cd /work/runs
140 : ```
141 :
142 : Copy over the input file
143 : `/work/spectre/tests/InputFiles/ExportCoordinates/InputTimeDependent3D.yaml`
144 : into your `/work/runs` directory. To run the executable, do
145 :
146 : ```
147 : spectre run --no-schedule -j 4 InputTimeDependent3D.yaml
148 : ```
149 :
150 : This will run it on 4 cores (the `--no-schedule` means it will run on the
151 : login/head node if you are using an HPC system). After this finishes you should
152 : see two `H5` files in your run directory:
153 :
154 : 1. ExportCoordinates3DVolume0
155 : 2. ExportCoordinates3DReductions
156 :
157 : The `Volume` file is where we store data from every element in our domain, like
158 : the coordinates or the metric. The `Reductions` file is for more global
159 : quantities like the minimum grid spacing over all the elements in our domain.
160 :
161 : \note Next time you run the executable, you will have to either move or delete
162 : the existing `H5` files as SpECTRE will error if it detects that an output file
163 : already exists. This is to prevent you from accidentally overwriting data.
164 : You can also use the `--force / -f` and `--clean-output / -C` flags to have
165 : `spectre run` delete the existing files before running the executable.
166 :
167 : ## Visualizing our BBH Coordinates
168 :
169 : Now it's time to use Paraview to visualize the coordinates we use for our BBH
170 : evolutions! SpECTRE will actually export the physical frame coordinates for
171 : every executable we have because they are a really useful diagnostic to have. We
172 : are just using the ExportCoordinates executable here so that you don't have to
173 : run a BBH evolution on your laptop which probably wouldn't work because of
174 : memory requirements.
175 :
176 : Before we get to Paraview, we have to tell paraview how to actually use the
177 : coordinates in the `Volume` `H5` file. To do this we have a tool called
178 : `generate-xdmf` in our Python command-line interface. Inside the `runs`
179 : directory where you have the `H5` files, run
180 :
181 : ```
182 : spectre generate-xdmf \
183 : --subfile-name element_data --output BBH_Coords \
184 : ExportCoordinates3DVolume*h5
185 : ```
186 :
187 : We output volume data per node so we append the
188 : node number to each volume file we have. Since you're most likely running on a
189 : laptop, you'll only be running on one node so you should only get one output
190 : file for the volume. The `--subfile-name` argument is the group name inside the
191 : `H5` file where the data is stored (groups can be checked by
192 : `h5ls -r FILE_NAME`). `generate-xdmf` will generate a file called
193 : `BBH_Coords.xmf`. Make sure to keep this `.xmf` file next to the volume file it
194 : was generated from. It uses relative paths to find the volume file which means
195 : if you move it, you won't be able to visualize anything.
196 :
197 : ### Attaching Paraview
198 :
199 : This is where we actually need Paraview. We have a headless (no GUI) vesion of
200 : paraview inside the container which we will refer to as the "server". To start
201 : the Paraview server, run
202 :
203 : ```
204 : pvserver &
205 : ```
206 :
207 : \note For cluster users, check cluster-specific instructions for ParaView.
208 : You may require MPI and specify which port to use.
209 : You can run `mpirun -np 1 pvserver -p PARAVIEW_REMOTE_PORT` on the remote
210 : cluster. Here, `PARAVIEW_REMOTE_PORT` is the port that the ParaView server
211 : will use on the cluster. You may choose it to be, for example, 11112.
212 : With 'pvserver' running on the remote machine, open a new local terminal.
213 : Locally, run
214 : `ssh -L11111:YOUR_CLUSTER:PARAVIEW_REMOTE_PORT USERNAME@YOUR_CLUSTER`.
215 :
216 : The `&` is so that the server runs in the background. If you hit `Enter`
217 : a couple times you'll get back to being able to type commands. You should see
218 : some output similar to
219 :
220 : ```
221 : Waiting for client...
222 : Connection URL: cs://92bbb69f2af2:11111
223 : Accepting connection(s): 92bbb69f2af2:11111
224 : ```
225 :
226 : This means it's waiting for you to connect some external Paraview session (the
227 : "client") to the server. Now, ***outside*** the container, start a session of
228 : Paraview 5.10.1. (Again, you must use this version otherwise it won't work
229 : properly.)
230 :
231 : \note For cluster users, run `mpirun -n 1 pvserver --version` to check ParaView
232 : version. On your computer, download ***exactly the same*** version of Paraview.
233 :
234 : Go to `File > Connect`. Click `Add Server`. Name it whatever you
235 : want, but keep the Host as `localhost`, the Server Type as `Client/Server`, the
236 : Port as `11111` (remember the `-p 11111:11111` flag from the docker command?).
237 : Here's a snapshot of what it should look like before you configure.
238 :
239 : \image html paraview_server.png "Paraview server settings"
240 :
241 : Hit `Configure`, then hit `Save` (we don't care about the launch configuration).
242 : Now you should see a list of your configured servers. Select the one you just
243 : created and hit `Connect`. It may take a minute or two to connect to the server,
244 : but once you do on the left you'll see something like
245 :
246 : \image html paraview_connect.png "Successfully connected Paraview to a server"
247 :
248 : \note If you close your client, the server will stop and you won't be able to
249 : reconnect. You'll have to restart the server in the container.
250 :
251 : ### Open the XMF File in Paraview Client {#open_xmf}
252 :
253 : Now that you have Paraview connected to the container, open the `BBH_Coords.xmf`
254 : file you just generated inside Paraview (the paths you'll see are the ones in
255 : the container, not your filesystem). You may be prompted to choose which XDMF
256 : reader to use. Choose the `XDMF Reader` option. The `Xdmf3` options won't work.
257 : Once you choose a reader, on the left, you'll see
258 :
259 : \image html beginners_paraview_left.png "Paraview side-bar"
260 :
261 : You can uncheck all the boxes in the `Point Arrays` section as they aren't
262 : necessary for visualizing the coordinates. Then hit `Apply`. Now you should see
263 : a solid sphere. This isn't super helpful. In the top bar you should see a
264 : dropdown to change the style that the points are plotted in. Select `Surface
265 : With Edges` like so. (Note: Your top bar may look slightly different from this
266 : depending on what version of `Paraview` you have.)
267 :
268 : \image html beginners_paraview_top.png "Paraview top-bar"
269 :
270 : Now you'll have a solid sphere with highlighted lines. To view the interior of
271 : the domain, you'll need to add a filter. To access the filters, navigate to
272 : `Filters` on the top menu bar, hover over `Alphabetical`, and search for your
273 : filter of choice. Probably the two most helpful filters
274 : for viewing the domain are the `Slice` and `Clip` filters. (Note that you'll
275 : have to choose the `Surface With Edges` option for each filter separately.)
276 :
277 : `Slice` is fairly self explanatory in that it will show you a single plane
278 : through the domain. Experiment with different planes to see our whole domain
279 : structure!
280 :
281 : The `Clip` filter will remove all points "above" a certain plane, where "above"
282 : is in the direction of the normal of that plane. If you combine two orthogonal
283 : `Clip`s, you can actually view a 3D wedge of our domain. Try moving the centers
284 : of the planes to view the domain around our excision surfaces! They have a lot
285 : of cool structure.
286 :
287 : If you'd like to read more about our BBH domain, you can look at the
288 : documentation for `domain::creators::BinaryCompactObject`.
289 :
290 : ## Evolution of BBH Coordinates
291 :
292 : Now that you are able to export and visualize our BBH domain coordinates at a
293 : single time, let's make a small movie of the coordinates as they evolve! To do
294 : this, we'll need to edit the input file `InputTimeDependent3D.yaml`. If you
295 : aren't familiar with YAML, it's a file type that uses key-value pairs to create
296 : actual objects in our C++ code. Feel free to experiment with keys and values in
297 : our input files. If you're unsure about what a key or value should be, we offer
298 : an easy way to check the options in the input file without running a whole
299 : simulation. In your `/work/runs` directory, if you run
300 :
301 : ```
302 : spectre validate InputTimeDependent3D.yaml
303 : ```
304 :
305 : the executable will parse and check the input file. If you made a typo, or added
306 : an incorrect key/value, a list of the available keys and their associated values
307 : will be printed.
308 :
309 : To change the number of times we output the coordinates, we'll need to go to the
310 : `%EventsAndTriggers:` block of the input file. This block is mainly where we
311 : specify which quantities we want to observe in a simulation or where we
312 : "Trigger" a specific "Event" to happen. (For more info on `%EventsAndTriggers`,
313 : see the \ref tutorial_events_and_triggers tutorial.) Currently in this input
314 : file we only have one Trigger/Event pair. The %Trigger is `TimeCompares:` and
315 : the %Event is `Completion`. To have the simulation run longer, change the
316 : `Value:` under `TimeCompares:` to something larger. If you look at the
317 : `Evolution:` block above the `%EventsAndTriggers:` block, you'll see that the
318 : initial time step is `0.5`. The way this executable is set up, the coordinates
319 : will be exported every time step. So set the final time `Value:` under
320 : `TimeCompares:` to some larger multiple of `0.5` so that you'll have the
321 : coordinates at a bunch of different times (a final time of `20` is reasonable.
322 : Depending on how many cores you run on this should take a couple minutes).
323 :
324 : Then, run the executable just like you did above (remember to move or delete the
325 : existing `H5` files), run `generate-xdmf`, and open it in Paraview and apply
326 : some filters of your choice. We recommend using a `Slice` filter with the normal
327 : pointing in the `-z` direction. This is because our BBH domain rotates about the
328 : `z` axis. Now, in the top bar of Paraview, you should see a "Play" button that
329 : looks like a sideways triangle (see the second image in the \ref open_xmf
330 : section). If you click this, Paraview will step through all the timesteps in the
331 : output files and you'll be able to see the domain rotate a bit!
332 :
333 : Next, we encourage you to play with the other inputs that control how the domain
334 : evolves over time. These options are housed in the
335 :
336 : ```yaml
337 : DomainCreator:
338 : BinaryCompactObject:
339 : ...
340 : TimeDependentMaps:
341 : ExpansionMap:
342 : ...
343 : RotationMap:
344 : ...
345 : SizeMapA:
346 : ...
347 : SizeMapB:
348 : ...
349 : ```
350 :
351 : block of the input file. Since this tutorial is more about running the code, we
352 : won't go into too much detail about each option. However, in general:
353 :
354 : 1. `ExpansionMap` is a global map (all parts of the domain) that controls the
355 : separation between the excision surfaces
356 : 2. `RotationMap` is a global map that controls how the excision spheres rotate
357 : about each other
358 : 3. `SizeMap` is a local map only around the excision spheres (not in the wave
359 : zone) that control the compression of grid points.
360 :
361 : Play around with these values! You may get an error if you put something that's
362 : too unphysical, but this is a fairly consequence-free playground for you to
363 : explore so just try a different value.
364 :
365 : Now you have a movie of how BBH coordinates evolve in a SpECTRE simulation!
366 :
367 : ## Exploring DG+FD
368 :
369 : Now that you are able to run, and visualize SpECTRE, let's explore a feature
370 : that is fairly unique to SpECTRE and is really powerful for handling
371 : discontinuities and shocks in our simulations. We call this feature `DG+FD`
372 : (it's also sometimes referred to as just `subcell`).
373 :
374 : ### Description of DG+FD
375 :
376 : `FD` is the usual finite difference you are used to. All of the BSSN codes use
377 : finite difference for solving Einstein's equations. FD is very good at capturing
378 : shocks and discontinuities and is a very robust method, making it well suited
379 : for hydro problems and other systems that have shocks and discontinuities.
380 :
381 : `DG` stands for Discontinuous Galerkin. DG is a spectral method for representing
382 : a solution on a grid, meaning that instead of taking the difference between the
383 : function value at two points to get the derivative, it uses known basis
384 : functions to represent the solution. Then the derivative can be known
385 : analytically and you only need to supply the coefficients for the basis. DG
386 : works best for representing smooth solutions; ones with very few shocks and
387 : discontinuities (like GR in vacuum). This makes DG much faster than FD for
388 : smooth solutions.
389 :
390 : In SpECTRE, we combine these two different methods into one system to take
391 : advantage of the strengths of each. When we have a solution that is smooth in
392 : some parts of the domain, but has shocks in other parts, using only one of these
393 : methods has disadvantages. If we only used DG, we wouldn't be able to resolve
394 : the shocks very well driving the errors up a lot. If we only used FD, we'd be
395 : able to represent the solution well, but it would be computationally
396 : inefficient. So we combine DG+FD so that we only do DG in the parts of the
397 : domain where the solution is smooth, and switch to FD in parts where there may
398 : be a shock or discontinuity. The algorithm for switching between DG and FD is
399 : explained in this image.
400 :
401 : \image html dg_fd_schematic.png "Scheme for switching between DG and FD (credit: Nils Deppe)"
402 :
403 : If you'd like to learn more about how SpECTRE implements its DG+FD scheme, you
404 : can read [the paper](https://arxiv.org/abs/2109.11645) on the ArXiv.
405 :
406 : ### Running the Kuzmin Problem
407 :
408 : To demonstrate DG+FD, we will be evolving the \link
409 : ScalarAdvection::Solutions::Kuzmin Kuzmin \endlink problem using the
410 : `EvolveScalarAdvection2D` executable. This is a simple test problem that
411 : rotates a set of geometric shapes with uniform angular velocity, which can be
412 : used to evaluate how well a numerical code can handle discontinuities stably
413 : over time. Inside the container make a new directory `/work/runs2` where you
414 : will run it. Also copy the default input file in
415 : `/work/spectre/tests/InputFiles/ScalarAdvection/Kuzmin2D.yaml` to this new
416 : `/work/runs2` directory.
417 :
418 : ### Changing the Default Input File
419 :
420 : The default input file has very low resolution so we'll need to crank that up a
421 : bit. The way to do this is to change the initial refinement levels and initial
422 : number of grid points which are located in
423 :
424 : ```yaml
425 : DomainCreator:
426 : Rectangle:
427 : ...
428 : InitialRefinement: [x, y]
429 : InitialGridPoints: [x, y]
430 : ```
431 :
432 : `InitialRefinement:` represents how many times we split a `Block` in half in
433 : order to create `Element`s, which are the fundamental units of our domain. So an
434 : initial refinement of `[1, 1]` means we split a single Block into 4 elements
435 : (split in half once in each direction). For an initial refinement of `[2, 2]` we
436 : first do 1 refinement like before, and then split each of the resulting 4
437 : elements in half again in each direction, resulting in 16 total Elements. To
438 : determine the total number of Elements for a given refinement (same in all
439 : directions), just do $2^{\mathrm{Dim * Refinement}}$. If you're confused by
440 : the terminology we use to describe the domain, we have a \ref domain_concepts
441 : guide that explains all terms related to our domain.
442 :
443 : `InitialGridPoints` represents the number of grid points per dimension in each
444 : Element after the final refinement has been applied. So if we had an initial
445 : refinement of `[2, 2]` like above and then initial grid points `[3, 3]` in each
446 : Element, we'd have a total of 9x16=144 grid points.
447 :
448 : As for actual numbers to use, you can experiment to see what gives good,
449 : well-resolved results. You'll definitely need more refinement than the default
450 : input file, but since refinement scales exponentially, this can become very
451 : expensive very quickly. On a laptop, you probably shouldn't go higher than
452 : refinement `[6, 6]`. As for grid points, this will depend on how much refinement
453 : you have. If you have a ton of small elements, you won't need too many grid
454 : points to resolve the solution; something like `[4, 4]` would work. If you don't
455 : have a lot of refinement, you may want more grid points if you still want to
456 : resolve your solution. For a DG scheme, increasing the number of grid points (p
457 : refinement) reduces the numerical error exponentially where the solution is
458 : smooth, so computational resources are used more effectively. However, to
459 : resolve shocks and discontinuities we have to refine the domain into more and
460 : smaller elements instead (h refinement). Striking the most effective balance
461 : between h and p refinement in different parts of the domain is the job of an
462 : adaptive mesh refinement (AMR) algorithm.
463 :
464 : The default input file only runs for a few time steps so we'll want to make this
465 : run longer so we can actually see some evolution. From the documentation of the
466 : \link ScalarAdvection::Solutions::Kuzmin Kuzmin \endlink system, the solution
467 : will rotate with an angular velocity of `1.0` (in code units). Thus, to do a
468 : full orbit, it will take `6.28` code units of time. In the `%EventsAndTriggers:`
469 : block of the input file, we see that the `Completion` event is triggered by the
470 : `Slabs` trigger. We could, in theory, calculate out how many slabs `6.28` code
471 : units is using the time step, but that's super tedious. Instead let's trigger
472 : completion using the `TimeCompares` trigger instead. We used this before when
473 : exporting the BBH coordinates, so just copy over the yaml block and change the
474 : `Value:`.
475 :
476 : Your final `%EventsAndTriggers:` block should look something like this:
477 :
478 : ```yaml
479 : EventsAndTriggers:
480 : - Trigger:
481 : TimeCompares:
482 : Comparison: GreaterThanOrEqualTo
483 : Value: 6.28
484 : Events:
485 : - Completion
486 : ...
487 : ```
488 :
489 : Now you should be ready to run the executable and get some output. Here, you
490 : will almost definitely benefit by running this on many cores by adding the
491 : `-j N` flag to the command you use to run the executable. Since we use lots
492 : of smaller elements, we distribute these over the available resources via a
493 : \link domain::BlockZCurveProcDistribution space filling curve \endlink to speed
494 : things up.
495 :
496 : ```
497 : spectre run --no-schedule -j 4 Kuzmin2D.yaml
498 : ```
499 :
500 : ### Visualizing the Kuzmin Problem
501 :
502 : Once your run finishes, extract the volume data with `generate-xdmf` using
503 :
504 : ```
505 : spectre generate-xdmf \
506 : --subfile-name VolumeData --output kuzmin_problem \
507 : ScalarAdvectionKuzmin2DVolume*h5
508 : ```
509 :
510 : (Note that the `subfile-name` is different than before because it was different
511 : in the input file) and load it into Paraview once again. We are only interested
512 : in the quantity `U` which is the scalar field we were evolving. You can uncheck
513 : any other boxes. So now, instead of coordinates on your screen, you should see a
514 : large square colored by the solution profile described in the \link
515 : ScalarAdvection::Solutions::Kuzmin Kuzmin \endlink system. You should also
516 : notice that there are smaller squares that don't touch each other in the middle
517 : of the domain and on the edges there are large sections that are continuous.
518 : These are the FD and DG grids, respectively. If you go to the top bar in
519 : Paraview and change how you view the grid to `Surface With Edges`, this will
520 : become even more apparent.
521 :
522 : You will notice that the FD grid is mostly around where the interesting features
523 : are in the solution profile; the cylinder with a wedge cut out, the cone, and
524 : the hump. And then the DG grid is mostly where the solution should be zero
525 : towards the boundary of the domain (i.e. the very smooth part). So right from
526 : the start, you can see that we are saving computational effort by only doing the
527 : expensive, yet robust, method (FD) where it is necessary and the efficient
528 : method (DG) everywhere else where the solution is smooth.
529 :
530 : Now hit the "Play" button in the top bar of Paraview and watch the solution
531 : evolve. You'll notice that the elements in the domain switch back and forth
532 : between FD and DG. They do so in such a way that the elements will switch to FD
533 : when an interesting feature enters the element and then switch back to DG once
534 : the feature leaves. In this way, we are able to actually track shocks and
535 : discontinuities in real time in our solution by where the code switches to using
536 : FD instead of DG. This is extremely useful for expensive GRMHD simulations where
537 : we only want to do FD at a shock boundary, yet that shock boundary is moving
538 : through the domain. We are able to dynamically track this shock and resolve it
539 : well with FD, then switch back to DG after the shock passes through and the
540 : solution has settled down again.
541 :
542 : A pretty cool filter you can add is `Warp By Scalar`. In the left panel, choose
543 : the solution variable `U` as the scalar to use and hit `Apply`. In the viewing
544 : panel there should be a `2D` or `3D` button that you can toggle to make the view
545 : 3D. Once you do that you should be able to see that the height of the feature is
546 : your solution `U`. If you change the grid to `Surface With Edges` you can see
547 : the FD or DG grids warp with the solution. And if you hit "Play" it'll rotate
548 : around and you'll see the features moving in 3D! (Don't worry if you can't find
549 : this filter. Not all versions of Paraview may have it.)
550 :
551 : We encourage you to play around with the refinement and grid points before the
552 : next section to get a feel for how each changes the runtime and accuracy of
553 : solution.
554 :
555 : ## Editing the Kuzmin System
556 :
557 : Hopefully now you feel comfortable enough running SpECTRE that you
558 : can get the default input file for the pre-built executables, edit it, and run
559 : it. Now we are going to try our hand at actually editing some code in SpECTRE
560 : and then building SpECTRE. We're going to stick with the \link
561 : ScalarAdvection::Solutions::Kuzmin Kuzmin \endlink system and add a new feature
562 : to the solution profile!
563 :
564 : You can find the files for the Kuzmin system at
565 : `/work/spectre/src/PointwiseFunctions/AnalyticSolutions/ScalarAdvection/
566 : Kuzmin.?pp`.
567 : In the `hpp` file, you'll see a lot of Doxygen documentation and then the actual
568 : Kuzmin class. The only function that you will need to care about is
569 :
570 : ```cpp
571 : template <typename DataType>
572 : tuples::TaggedTuple<ScalarAdvection::Tags::U> variables(
573 : const tnsr::I<DataType, 2>& x, double t,
574 : tmpl::list<ScalarAdvection::Tags::U> /*meta*/) const;
575 : ```
576 :
577 : All of our analytic solutions have a function similar to this that will set the
578 : value corresponding to the tag in the return type. If you're unfamiliar with
579 : tags in SpECTRE, you can look at these sections for an explanation, \ref
580 : databox_a_taggedtuple_databox and \ref databox_a_proper_databox. However, it's
581 : basically just a fancy way of doing a compile-time key/value pair. The tag is
582 : the key, and the value is whatever you want it to be. In our case, the value is
583 : a tensor, representing the solution.
584 :
585 : The definition of this function in the `cpp` file is where you will be editing
586 : the actual Kuzmin solution. Towards the bottom of this function, there is a
587 : `for` loop that sets the solution at every grid point. This is where you will
588 : add in a new feature to the solution.
589 :
590 : You can pick any feature you want to add, so long as it's inside the domain
591 : bounds of `[0,1]x[0,1]` and centered around `(0.75, 0.5)`. This is because of
592 : how the kuzmin solution is set up with existing features at `(0.25, 0.5); (0.5,
593 : 0.25); (0.5, 0.75)`. If you're having trouble thinking of a feature to add try
594 : one of the following features:
595 :
596 : - Square centered at `(0.75, 0.5)` with solution value `1.0`
597 : - Side length `0.1` (any larger and it might interfere with the other
598 : features)
599 : - Circle of radius `0.045` centered on the square with value `0.0`
600 : - Triangle centered at `(0.75, 0.5)` with one corner facing in the `+x`
601 : direction with solution value `1.0`
602 : - Square centered at `(0.75, 0.5)`
603 : - Side length `0.1` (any larger and it might interfere with the other
604 : features)
605 : - Left half of the square has value `1.0` and right half of the square has
606 : value `0.5`
607 :
608 : \note The more detailed you make your feature, the more resolution you will need
609 : to resolve it.
610 :
611 : ### Re-building SpECTRE
612 :
613 : Once you have your feature coded up, go ahead and save your changes. Now we will
614 : build SpECTRE! Go to the `/work/spectre/build` directory. This is where you have
615 : to be in order to build SpECTRE. We use [CMake](https://cmake.org/) to configure
616 : our build directory. However, since the executables are already pre-built, this
617 : means the build directory is already configured! So you don't have to worry
618 : about `CMake` for now. If you wanted to reconfigure, for example using a
619 : different compiler, then you'd have to run `CMake`. If you want to learn more
620 : about how we use `CMake`, take a look at the \ref common_cmake_flags developers
621 : guide.
622 :
623 : To build the Kuzmin executable, run
624 :
625 : ```
626 : make EvolveScalarAdvection2D
627 : ```
628 :
629 : This should be very fast because you only edited a `cpp` file. Congrats! You've
630 : just built SpECTRE!
631 :
632 : Now re-run the executable in your `/work/runs2` directory. Hopefully everything
633 : works and you get some output. When you plot it in Paraview, it should look
634 : almost the same as before except your feature will be there too rotating with
635 : the others! How cool! You can also see if your feature needs FD or DG more by
636 : how much it switches back and forth.
637 :
638 : Experiment some more with either different features or different resolution!
639 :
640 : ## Conclusions
641 :
642 : Congrats! You've made it through the tutorial! If you only want to run our
643 : pre-built executables, you have all the tools necessary to run, visualize, and
644 : re-build them. If you want a full list of our executables, do
645 : `make list` in the build directory. This will also include our `Test_`
646 : executables which you can just ignore.
647 :
648 : In an already configured build directory, all you have to do to build a new
649 : executable is
650 :
651 : ```
652 : make ExecutableName
653 : ```
654 :
655 : and then you can copy the default input file from
656 : `/work/spectre/tests/InputFiles` and run it. Running an executable with the
657 : `--help` flag will give a description of what system is being evolved and the
658 : input options necessary.
|
