SpECTRE Documentation Coverage Report
Current view: top level - __w/spectre/spectre/docs/DevGuide - WritingTests.md Hit Total Coverage
Commit: f1ddee3e40d81480e49140855d2b0e66fafaa908 Lines: 0 1 0.0 %
Date: 2020-12-02 17:35:08
Legend: Lines: hit not hit

          Line data    Source code
       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # Writing Unit Tests {#writing_unit_tests}
       6             : 
       7             : Unit tests are placed in the appropriate subdirectory of `tests/Unit`, which
       8             : mirrors the directory hierarchy of `src`. The tests are all compiled into
       9             : individual libraries to keep link time of testing executables low. Typically
      10             : there should be one test library for each production code library. For example,
      11             : we have a `DataStructures` library and a `Test_DataStructures` library. When
      12             : adding a new test there are several scenarios that can occur, which are outlined
      13             : below.
      14             : 
      15             : - You are adding a new source file to an existing test library:<br>
      16             :   If you are adding a new source file in a directory that already has a
      17             :   `CMakeLists.txt` simply create the source file, which should be named
      18             :   `Test_ProductionCodeFileBeingTest.cpp` and add that to the `LIBRARY_SOURCES`
      19             :   in the `CMakeLists.txt` file in the same directory you are adding the `cpp`
      20             :   file.<br>
      21             :   If you are adding a new source file to a library but want to place it in a
      22             :   subdirectory you must first create the subdirectory. To provide a concrete
      23             :   example, say you are adding the directory `TensorEagerMath` to
      24             :   `tests/Unit/DataStructures`. After creating the directory you must add a call
      25             :   to `add_subdirectory(TensorEagerMath)` to
      26             :   `tests/Unit/DataStructures/CMakeLists.txt` *before* the call to
      27             :   `add_test_library` and *after* the `LIBRARY_SOURCES` are set. Next add the
      28             :   file `tests/Unit/DataStructures/TensorEagerMath/CMakeLists.txt`, which should
      29             :   add the new source files by calling `set`, e.g.
      30             :   ```
      31             :   set(LIBRARY_SOURCES
      32             :       ${LIBRARY_SOURCES}
      33             :       Test_ProductionCodeFileBeingTest.cpp
      34             :       PARENT_SCOPE)
      35             :   ```
      36             :   The `PARENT_SCOPE` flag tells CMake to make the changes visible in the
      37             :   CMakeLists.txt file that called `add_subdirectory`. You can now add the
      38             :   `Test_ProductionCodeFileBeingTested.cpp` source file.
      39             : - You are adding a new directory:<br>
      40             :   If the directory is a new lowest level directory you must add a
      41             :   `add_subdirectory` call to `tests/Unit/CMakeLists.txt`. If it is a new
      42             :   subdirectory you must add a `add_subdirectory` call to the
      43             :   `CMakeLists.txt` file in the directory where you are adding the
      44             :   subdirectory. Next you should read the part on adding a new test library.
      45             : - You are adding a new test library:<br>
      46             :   After creating the subdirectory for the new test library you must add a
      47             :   `CMakeLists.txt` file. See `tests/Unit/DataStructures/CMakeLists.txt` for
      48             :   an example of one. The `LIBRARY` and `LIBRARY_SOURCES` variables set the name
      49             :   of the test library and the source files to be compiled into it. The library
      50             :   name should be of the format `Test_ProductionLibraryName`, for example
      51             :   `Test_DataStructures`. The library sources should be only the source files in
      52             :   the current directory. The `add_subdirectory` command can be used to add
      53             :   source files in subdirectories to the same library as is done in
      54             :   `tests/Unit/CMakeLists.txt`. The `CMakeLists.txt` in
      55             :   `tests/Unit/DataStructures/TensorEagerMath` is an example of how to add source
      56             :   files to a library from a subdirectory of the library. Note that the setting
      57             :   of `LIBRARY_SOURCES` here first includes the current `LIBRARY_SOURCES` and at
      58             :   the end specifies `PARENT_SCOPE`. The `PARENT_SCOPE` flag tells CMake to
      59             :   modify the variable in a scope that is visible to the parent directory,
      60             :   i.e. the `CMakeLists.txt` that called `add_subdirectory`.<br>
      61             :   Finally, in the `CMakeLists.txt` of your new library you must call
      62             :   `add_test_library`. Again, see `tests/Unit/DataStructures/CMakeLists.txt` for
      63             :   an example. The `add_test_library` function adds a test library with the name
      64             :   of the first argument and the source files of the third argument. The second
      65             :   argument is the path of the library's directory relative to `tests/Unit`. For
      66             :   example, for `Test_DataStructures` it is simply `DataStructures`. The fourth
      67             :   and final argument to `add_test_library` are the libraries that must be
      68             :   linked. Typically this should only be the production library you're
      69             :   testing. For example, `Test_DataStructures` should specify only
      70             :   `DataStructures` as the library to link. If you are testing a header-only
      71             :   "library" then you do not link any libraries (they must be linked in by the
      72             :   libraries actually testing your dependencies). In this case the last argument
      73             :   should be `"" # Header-only, link dependencies included from testing lib`
      74             : 
      75             : All tests must start with
      76             : ```cpp
      77             : // Distributed under the MIT License.
      78             : // See LICENSE.txt for details.
      79             : 
      80             : #include "Framework/TestingFramework.hpp"
      81             : ```
      82             : The file `tests/Unit/Framework/TestingFramework.hpp` must always be the first
      83             : include in the test file and must be separated from the STL includes by a blank
      84             : line. All classes and free functions should be in an anonymous/unnamed
      85             : namespace, e.g.
      86             : ```cpp
      87             : namespace {
      88             : class MyFreeClass {
      89             :   /* ... */
      90             : };
      91             : 
      92             : void my_free_function() noexcept {
      93             :   /* ... */
      94             : }
      95             : }  // namespace
      96             : ```
      97             : This is necessary to avoid symbol redefinition errors during linking.
      98             : 
      99             : Test cases are added by using the `SPECTRE_TEST_CASE` macro. The first argument
     100             : to the macro is the test name, e.g. `"Unit.DataStructures.Tensor"`, and the
     101             : second argument is a list of tags. The tags list is a string where each element
     102             : is in square brackets. For example, `"[Unit][DataStructures]"`. The tags should
     103             : only be the type of test, in this case `Unit`, and the library being tested, in
     104             : this case `DataStructures`. The `SPECTRE_TEST_CASE` macro should be treated as a
     105             : function, which means that it should be followed by `{ /* test code */ }`. For
     106             : example,
     107             : \snippet Test_Tensor.cpp example_spectre_test_case
     108             : From within a `SPECTRE_TEST_CASE` you are able to do all the things you would
     109             : normally do in a C++ function, including calling other functions, setting
     110             : variables, using lambdas, etc.
     111             : 
     112             : The `CHECK` macro in the above example is provided by
     113             : [Catch2](https://github.com/catchorg/Catch2) and is used to check conditions. We
     114             : also provide the `CHECK_ITERABLE_APPROX` macro which checks if two `double`s or
     115             : two iterable containers of `double`s are approximately
     116             : equal. `CHECK_ITERABLE_APPROX` is especially useful for comparing `Tensor`s,
     117             : `DataVector`s, and `Tensor<DataVector>`s since it will iterate over nested
     118             : containers as well.
     119             : 
     120             : \warning Catch's `CHECK` statement only prints numbers out to approximately 10
     121             : digits at most, so you should generally prefer `CHECK_ITERABLE_APPROX` for
     122             : checking double precision numbers, unless you want to check that two numbers are
     123             : bitwise identical.
     124             : 
     125             : All unit tests must finish within a few seconds, the hard limit is 5, but having
     126             : unit tests that long is strongly discouraged. They should typically complete in
     127             : less than half a second. Tests that are longer are often no longer testing a
     128             : small enough unit of code and should either be split into several unit tests or
     129             : moved to an integration test.
     130             : 
     131             : #### Discovering New and Renamed Tests
     132             : 
     133             : When you add a new test to a source file or rename an existing test the change
     134             : needs to be discovered by the testing infrastructure. This is done by building
     135             : the target `rebuild_cache`, e.g. by running `make rebuild_cache`.
     136             : 
     137             : #### Testing Pointwise Functions
     138             : 
     139             : Pointwise functions should generally be tested in two different ways. The first
     140             : is by taking input from an analytic solution and checking that the computed
     141             : result is correct. The second is to use the random number generation comparison
     142             : with Python infrastructure. In this approach the C++ function being tested is
     143             : re-implemented in Python and the results are compared. Please follow these
     144             : guidelines:
     145             : 
     146             : - The Python implementation should be in a file with the same name as the source
     147             :   file that is being re-implemented and placed in the same directory as its
     148             :   corresponding `Test_*.cpp` source file.
     149             : - The functions should have the same names as the C++ functions they
     150             :   re-implement.
     151             : - If a function does sums over tensor indices then
     152             :   [`numpy.einsum`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.einsum.html)
     153             :   should be used in Python to provide an alternative implementation of the loop
     154             :   structure.
     155             : - You can import Python functions from other re-implementations in the
     156             :   `tests/Unit/` directory to reduce code duplication. Note that the path you
     157             :   pass to `pypp::SetupLocalPythonEnvironment` determines the directory from
     158             :   which you can import Python modules. Either import modules directly from the
     159             :   `tests/Unit/` directory (e.g. `import
     160             :   PointwiseFunction.GeneralRelativity.Christoffel as christoffel`) or use
     161             :   relative imports like `from . import Christoffel as christoffel`. Don't assume
     162             :   the Python environment is set up in a subdirectory of `tests/Unit/`.
     163             : - The python code is formatted according to the `.style.yapf` file in the root
     164             :   of the repository.
     165             : 
     166             : It is possible to test C++ functions that return by value and ones that return
     167             : by `gsl::not_null`. In the latter case, since it is possible to return multiple
     168             : values, one Python function taking all non-`gsl::not_null` arguments must be
     169             : supplied for each `gsl::not_null` argument to the C++. To perform the test the
     170             : `pypp::check_with_random_values()` function must be called. For example, the
     171             : following checks various C++ functions by calling into `pypp`:
     172             : 
     173             : \snippet Test_Pypp.cpp cxx_two_not_null
     174             : 
     175             : The corresponding Python functions are:
     176             : 
     177             : \snippet PyppPyTests.py python_two_not_null
     178             : 
     179             : #### Writing and Fixing Random-Value Based Tests
     180             : 
     181             : Many tests in SpECTRE make use of randomly generated numbers in order to
     182             : increase the parameter space covered by the tests. The random number generator
     183             : is set up using:
     184             : ```cpp
     185             : MAKE_GENERATOR(gen);
     186             : ```
     187             : The generator `gen` can then be passed to distribution classes such as
     188             : `std::uniform_real_distribution` or `UniformCustomDistribution`.
     189             : 
     190             : Each time the test is run, a different random seed will be used.  When writing a
     191             : test that uses random values, it is good practice to run the test at least
     192             : \f$10^4\f$ times in order to set any tolerances on checks used in the test.
     193             : This can be done by using the following command in the build directory
     194             : (SPECTRE_BUILD_DIR):
     195             : ```
     196             : ctest --repeat-until-fail 10000 -R TEST_NAME
     197             : ```
     198             : where `TEST_NAME` is the test name passed to `SPECTRE_TEST_CASE`
     199             : (e.g. `Unit.Evolution.Systems.CurvedScalarWave.Characteristics`).
     200             : 
     201             : If a test case fails when using a random number generated by `MAKE_GENERATOR`,
     202             : as part of the output from the failed test will be the text
     203             : ```
     204             : Seed is:  SEED from FILE_NAME:LINE_NUMBER
     205             : ```
     206             : Note that the output of tests can be found in
     207             : `SPECTRE_BUILD_DIR/Testing/Temporary/LastTest.log`
     208             : 
     209             : The failing test case can then be reproduced by changing `MAKE_GENERATOR` call
     210             : at the provided line in the given file to
     211             : ```cpp
     212             : MAKE_GENERATOR(gen, SEED);
     213             : ```
     214             : If the `MAKE_GENERATOR` is within `CheckWithRandomValues.hpp`, the failing test
     215             : case most likely has occurred within a call to
     216             : `pypp::check_with_random_values()`.  In such a case, additional information
     217             : should have been printed to help you determine which call to
     218             : `pypp::check_with_random_values()` has failed.  The critical information is
     219             : the line
     220             : ```
     221             : function:  FUNCTION_NAME
     222             : ```
     223             : where `FUNCTION_NAME` should correspond to the third argument of a call to
     224             : `pypp::check_with_random_values()`.  The seed that caused the test to fail can
     225             : then be passed as an additional argument to `pypp::check_with_random_values()`,
     226             : where you may also need to pass in the default value of the comparison
     227             : tolerance.
     228             : 
     229             : Typically, you will need to adjust a tolerance used in a `CHECK` somewhere in
     230             : the test in order to get the test to succeed reliably.  The function
     231             : `pypp::check_with_random_values()` takes an argument that specifies the lower
     232             : and upper bounds of random quantities.  Typically these should be chosen to be
     233             : of order unity in order to decrease the chance of occasionally generating large
     234             : numbers through multiplications which can cause an error above a reasonable
     235             : tolerance.
     236             : 
     237             : #### Testing Failure Cases
     238             : 
     239             : Adding the "attribute" `// [[OutputRegex, Regular expression to
     240             : match]]` before the `SPECTRE_TEST_CASE` macro will force ctest to only
     241             : pass the particular test if the regular expression is found in the
     242             : output of the test. This can be used to test error handling. When
     243             : testing `ASSERT`s you must mark the `SPECTRE_TEST_CASE` as
     244             : `[[noreturn]]`, add the macro `ASSERTION_TEST();` to the beginning of
     245             : the test, and also have the test call `ERROR("Failed to trigger ASSERT
     246             : in an assertion test");` at the end of the test body.  The test body
     247             : should be enclosed between `#%ifdef SPECTRE_DEBUG` and an `#%endif`
     248             : For example,
     249             : 
     250             : \snippet Test_AssertAndError.cpp assertion_test_example
     251             : 
     252             : If the `#%ifdef SPECTRE_DEBUG` block is omitted then compilers will
     253             : correctly flag the code as being unreachable which results in
     254             : warnings.
     255             : 
     256             : You can also test `ERROR`s inside your code. These tests need to have
     257             : the `OutputRegex`, and also call `ERROR_TEST();` at the
     258             : beginning. They do not need the `#%ifdef SPECTRE_DEBUG` block, they
     259             : can just call have the code that triggers an `ERROR`. For example,
     260             : 
     261             : \snippet Test_AssertAndError.cpp error_test_example
     262             : 
     263             : Note that a `OutputRegex` can also be specified in a test that is
     264             : supposed to succeed with output that matches the regular expression.
     265             : In this case, the first line of the test should call the macro
     266             : `OUTPUT_TEST();`.
     267             : 
     268             : ### Testing Actions
     269             : 
     270             : The action testing framework is documented as part of the `ActionTesting`
     271             : namespace.
     272             : 
     273             : ### Building and Running A Single Test File
     274             : 
     275             : In cases where low-level header files are frequently being altered and the
     276             : changes need to be tested, building `RunTests` becomes extremely time
     277             : consuming. The `RunSingleTest` executable in `tests/Unit/RunSingleTest` allows
     278             : one to compile only a select few of the test source files and only link in the
     279             : necessary libraries. To set which test file and libraries are linked into
     280             : `RunSingleTest` edit the `tests/Unit/RunSingleTest/CMakeLists.txt`
     281             : file. However, do not commit your changes to that file since it is meant to
     282             : serve as an example. To compile `RunSingleTest` use `make RunSingleTest`, and to
     283             : run it use `BUILD_DIR/bin/RunSingleTest Unit.Test.Name`.
     284             : 
     285             : \warning
     286             : `Parallel::abort` does not work correctly in the `RunSingleTest` executable
     287             : because a segfault occurs inside Charm++ code after the abort message is
     288             : printed.

Generated by: LCOV version 1.14