Writing Unit Tests

Unit tests are placed in the appropriate subdirectory of tests/Unit, which mirrors the directory hierarchy of src. The tests are all compiled into individual libraries to keep link time of testing executables low. Typically there should be one test library for each production code library. For example, we have a DataStructures library and a Test_DataStructures library. When adding a new test there are several scenarios that can occur, which are outlined below.

All tests must start with

// Distributed under the MIT License.
// See LICENSE.txt for details.

The file tests/Unit/TestingFramework.hpp must always be the first include in the test file and must be separated from the STL includes by a blank line. All classes and free functions should be in an anonymous/unnamed namespace, e.g.

namespace {
class MyFreeClass {
/* ... */
};
void my_free_function() noexcept {
/* ... */
}
} // namespace

This is necessary to avoid symbol redefinition errors during linking.

Test cases are added by using the SPECTRE_TEST_CASE macro. The first argument to the macro is the test name, e.g. "Unit.DataStructures.Tensor", and the second argument is a list of tags. The tags list is a string where each element is in square brackets. For example, "[Unit][DataStructures]". The tags should only be the type of test, in this case Unit, and the library being tested, in this case DataStructures. The SPECTRE_TEST_CASE macro should be treated as a function, which means that it should be followed by { /* test code */ }. For example,

SPECTRE_TEST_CASE("Unit.DataStructures.Tensor.Frames",
"[Unit][DataStructures]") {
CHECK("Logical" == get_output(Frame::Logical{}));
CHECK("Grid" == get_output(Frame::Grid{}));
CHECK("Inertial" == get_output(Frame::Inertial{}));
CHECK("Distorted" == get_output(Frame::Distorted{}));
CHECK("NoFrame" == get_output(Frame::NoFrame{}));
}

From within a SPECTRE_TEST_CASE you are able to do all the things you would normally do in a C++ function, including calling other functions, setting variables, using lambdas, etc.

The CHECK macro in the above example is provided by Catch2 and is used to check conditions. We also provide the CHECK_ITERABLE_APPROX macro which checks if two doubles or two iterable containers of doubles are approximately equal. CHECK_ITERABLE_APPROX is especially useful for comparing Tensors, DataVectors, and Tensor<DataVector>s since it will iterate over nested containers as well.

Warning
Catch's CHECK statement only prints numbers out to approximately 10 digits at most, so you should generally prefer CHECK_ITERABLE_APPROX for checking double precision numbers, unless you want to check that two numbers are bitwise identical.

All unit tests must finish within a few seconds, the hard limit is 5, but having unit tests that long is strongly discouraged. They should typically complete in less than half a second. Tests that are longer are often no longer testing a small enough unit of code and should either be split into several unit tests or moved to an integration test.

Discovering New and Renamed Tests

When you add a new test to a source file or rename an existing test the change needs to be discovered by the testing infrastructure. This is done by building the target rebuild_cache, e.g. by running make rebuild_cache.

Testing Pointwise Functions

Pointwise functions should generally be tested in two different ways. The first is by taking input from an analytic solution and checking that the computed result is correct. The second is to use the random number generation comparison with python infrastructure. In this approach the C++ function being tested is re-implemented in python and the results are compared. If the function does sums over tensor indices then the einsum package should be used in python to provide an alternative implementation of the loop structure. The python implementations should be in a file name TestFunctions.py in the same directory as the Test_*.cpp source file and have the same name as the C++ function being tested. It is possible to test C++ functions that return by value and ones that return by gsl::not_null. In the latter case, since it is possible to return multiple values, one python function taking all non-gsl::not_null arguments must be supplied for each gsl::not_null argument to the C++. To perform the test the pypp::check_with_random_values() function must be called. For example, the following checks various C++ functions by calling into pypp:

pypp::check_with_random_values<2>(
&check_double_not_null2_scalar<Scalar<DataVector>>, "PyppPyTests",
{"check_double_not_null2_result0", "check_double_not_null2_result1"},
{{{0.0, 10.0}, {-10.0, 0.0}}}, scalar_dv);

The corresponding python functions are:

def check_double_not_null2_result0(t0, t1):
return np.sqrt(t0) + 1.0 / np.sqrt(-t1)
def check_double_not_null2_result1(t0, t1):
return 2.0 * t0 + t1

Testing Failure Cases

Adding the "attribute" // [[OutputRegex, Regular expression to match]] before the SPECTRE_TEST_CASE macro will force ctest to only pass the particular test if the regular expression is found. This can be used to test error handling. When testing ASSERTs you must mark the SPECTRE_TEST_CASE as [[noreturn]], add the macro ASSERTION_TEST(); to the beginning of the test, and also have the test call ERROR("Failed to trigger ASSERT in an assertion test"); at the end of the test body. For example,

// [[OutputRegex, Must copy into same size]]
[[noreturn]] SPECTRE_TEST_CASE("Unit.DataStructures.DataVector.ref_diff_size",
"[DataStructures][Unit]") {
#ifdef SPECTRE_DEBUG
DataVector data{1.43, 2.83, 3.94, 7.85};
DataVector data_ref;
data_ref.set_data_ref(data);
DataVector data2{1.43, 2.83, 3.94};
data_ref = data2;
ERROR("Failed to trigger ASSERT in an assertion test");
#endif
}

If the ifdef SPECTRE_DEBUG is omitted then compilers will correctly flag the code as being unreachable which results in warnings.

You can also test ERRORs inside your code. These tests need to have the OutputRegex, and also call ERROR_TEST(); at the beginning. The do not need the ifdef SPECTRE_DEBUG block, they can just call have the code that triggers an ERROR. For example,

// [[OutputRegex, 'a == b' violated!]]
[[noreturn]] SPECTRE_TEST_CASE(
"Unit.ErrorHandling.AbortWithErrorMessage.Assert",
"[Unit][ErrorHandling]") {
abort_with_error_message("a == b", __FILE__, __LINE__,
static_cast<const char*>(__PRETTY_FUNCTION__),
"Test Error");
}

Building and Running A Single Test File

In cases where low-level header files are frequently being altered and the changes need to be tested, building RunTests becomes extremely time consuming. The RunSingleTest executable in tests/Unit/RunSingleTest allows one to compile only a select few of the test source files and only link in the necessary libraries. To set which test file and libraries are linked into RunSingleTest edit the tests/Unit/RunSingleTest/CMakeLists.txt file. However, do not commit your changes to that file since it is meant to serve as an example. To compile RunSingleTest use make RunSingleTest, and to run it use BUILD_DIR/bin/RunSingleTest Unit.Test.Name.

Warning
Parallel::abort does not work correctly in the RunSingleTest executable because a segfault occurs inside Charm++ code after the abort message is printed.