Testing Framework
Using a testing framework for testing C++ code is not a requirement, but avoiding using one is like trying to avoid using the standard library – it’s possible, but why reinvent the wheel? Using a testing framework definitely speeds up the process, so I will use one here as well. There’s a number of testing frameworks available, but one stands out as the most popular and widely used and comes bundled with a mocking framework. I’m speaking of googletest, of course. It has been recently updated to use and require C++11 features (finally). I will be using the most recent release – 1.10.
Note that there are some fantastic alternatives to googletest available. Most notably Catch2 and doctest. Both are fully featured testing frameworks, that could be argued to be better, and definitely more “modern” than googletest. They lack, however, any support for mocking at this moment. Most often they’re used with standalone mocking frameworks like FakeIt or HippoMocks. Unfortunately, these seem to be unmaintained for some years now. Given these downsides, and one more important advantage of googletest – popularity in commercial projects – my choice goes to googletest. I’d still encourage you to check out the mentioned frameworks in your spare time.
No matter which framework we choose, we’ll be introducing an external dependency into our project. It will need to be managed somehow. Let’s see what choices are available.
Managing dependencies
Dependency management is another huge topic in and of itself. Here I will just mention some broad alternatives available for handling external dependencies in a C++ project. The first decision to make is between introducing the dependency as an already precompiled binary or compiling the dependency as part of our project. Both have some advantages and disadvantages. Let’s briefly look at both.
Choosing to introduce the dependency in the form of an already compiled library is often the first choice. It may seem trivial at first. In a CMakeLists.txt this would boil down to
find_package(SomeDependency)Simple enough. Right?
Kind off. But how does CMake find this dependency? Well, it searches some predefined set of paths and looks for CMake modules or config-files. Right… I encourage you to check out the documentation for now, and as always, I will return to this topic in the future. Long story short is that this dependency will need to be provided somehow. So again – we have some choises to make. Do we use a system package manager like apt, pacman, yum, or whatever else your system offers, to install the dependency system-wide (an option Windows users don’t really have)? Do we just download a tar.gz or .zip and extract it somewhere find_package can find it? Either way, these pre-requisites need to be met manually or ensured with an additional layer of scripting. Sounds like a lot of complexity. And it is. Just wait until you have to deal with linker errors.
Despite all of this complexity, this is often the only valid option for large projects. It has one deciding advantage – the dependencies don’t need to be recompiled over and over again. In projects with lots and lots of dependencies, this saves a ton of time.
The second major alternative is to compile the dependency alongside our project. This means we’d be pulling in the external dependencie’s sources somehow and adding them into the project. This could be done using something like git submodules, or even by copying the dependency into a dedicated subdirectory – an extremely simple solution, that may actually be valid in some circumstances (heresy I know, but for throw-away kind of code, why not?). These solutions however again rely on some external prerequisites (cloning the submodule). CMake offers a better alternative – the dependencies can be fully managed by the build system using either ExternalProject or FetchContent modules. The former is older and more powerfull, the latter builds on the first one and is much easier to use. ExternalProject is the solution recommended in googletest’s documentation. However, it describes a use that mimics what FetchContent does in a much simpler way, so for the sake of simplicity I will use the FetchContent module here.
For completeness, sake let me also mention that a more and more popular alternative is to use a package manager – a feature missing until recently from the C++ ecosystem. The only valid available choices right now are Conan and Microsoft’s vcpkg. Both are actively developed and aim to solve all the issues described above, and more. This is slowly becoming the standard approach but is too big of a topic for this post (Yes, I will be returning to this in the future).
So. FetchContent it is. Let’s see how to introduce googletest into a project using this module.
FetchContent Module
FetchContent is a module that ships with CMake. Its purpose is to download (or otherwise “fetch”) arbitrary external content for the purposes of incorporating it into a project. The content could be another C/C++ project that already uses CMake as its build system (this is the case with googletest), but it doesn’t have to be. This might as well be a project that doesn’t use CMake at all, it could be some arbitrary scripts – like the ones created in the previous part, or whatever else, that’s managed externally and needs to be brought into the project.
In our case googletest already supports CMake. This is the simplest use-case for FetchContent.
Since this is a library we will be using for testing let’s now introduce a subdirectory dedicated to this purpose. In the root directory of our project create a tests directory. And within that subdirectory create a CMakeLists.txt. The project root directory tree now looks as follows:
.
├── add
├── cmake
├── CMakeLists.txt
├── main.cpp
├── maths
├── subtract
└── tests
└── CMakeLists.txtOf course the tests subdirectory needs to be added in the top-level CMakeLists.txt using add_subdirectory.
The tests/CMakeLists.txt contains the following code:
include(FetchContent)
FetchContent_Declare(googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG release-1.10.0
)
FetchContent_MakeAvailable(googletest)And that’s it. Googletest will now be built as part of the project, with all the targets it defines being available for use. Let’s look at what’s going on here.
There are two major steps to using FetchContent. The first one is to declare an external dependency. Some form of source of the dependency needs to be specified. Here, this is a git repository and a specific tag. It could also be a URL or an SVN repository.
The second step is to process the declared content somehow. FetchContent documentation also calls this “populating” the target. Here a simplified form is used – FetchContent_MakeAvailable – it encapsulates all the complexity. Simplifying a little, it just checks internally if this target has already been processed (populated) and calls add_subdirectory on it, if it has not.
This two-step process takes into account the possibility of multiple such dependencies depending on each other. It would then handle the mutual dependency in a way that removes the redundancy (or potential versioning ambiguity) – fetching it only once. For example – let’s say that FetchContent is used to add two depedencies – A and B. But both A and B depend on C. FetchContent would then automatically handle the transitive dependency C, fetching it only once and making it available to both A and B. Neat.
This setup is not perfect. It might be cleaner to encapsulate the dependency-management code into a dedicated script or subdirectory, planning already to introduce more dependencies in the future. But following the YAGNI principle let’s just follow the most direct solution and refactor in the future, if necessary.
One more thing. As is, the FetchContent will verify if the declared dependencies are up to date everytime you build the project. This can be time consuming, esepcially once more dependencies are added. To prevent it from doing so add the following line after the FetchContent_MakeAvailable call
set(FETCHCONTENT_UPDATES_DISCONNECTED ON CACHE BOOL "")Now that googletest is available we can move on to writing some test cases. CMake won’t help us with that, but it can help us with managing and executing the tests. This is done using CTest – an integral component of CMake. And since we decided to use googletest it offers some additional convenience features.
CTest
Every CMake installation also includes CTest – a comprehensive utility for test execution and management. It actually offers quite a bit more and is designed to work with CDash – yet another tool in the CMake family – a testing dashboard, used to aggregate and display test results. I will not be discussing it in this guide.
Back to CTest. In day-to-day use CTest is a test driver utility for tests declared in the CMake build system. Its use can be enabled simply by adding one of the following lines in the top level CMakeLists.txt:
enable_testing()
# or
include(CTest)The difference here is that calling enable_testing will only enable the basic use of the ctest command line tool, while including the CTest module will additionally enable integration with CDash and define a BUILD_TESTING option, which can be used to conditionally build (or skip) tests. We don’t need the CDash integration, but the option we get for free can be useful, so let’s go with include(CTest) instead of enable_testing.
The simplest use of the BUILD_TESTING variable is to add the tests subdirectory only if BUILD_TESTING is True:
if (BUILD_TESTING)
add_subdirectory(tests)
endif()Note that BUILD_TESTING is True by default, if a user wishes to skip building tests, the flag must explicitly be set to OFF/False.
Now that testing is enabled, tests can be registered with CMake using the add_test command. To demonstrate its use, we will need to actually write some tests. Assume we have written the following trivial unit tests in a ut_maths.cpp file within the tests directory:
#include <gtest/gtest.h>
#include <maths/maths.h>
TEST(maths, addTest)
{
Maths::calc_status sc{};
const int result = Maths::add(42, 11, sc);
EXPECT_EQ(sc, Maths::calc_status::success);
EXPECT_EQ(result, 42 + 11);
}
TEST(maths, subTest)
{
Maths::calc_status sc{};
const int result = Maths::subtract(42, 11, sc);
EXPECT_EQ(sc, Maths::calc_status::success);
EXPECT_EQ(result, 42 - 11);
}Sure, these don’t actually test much, but that’s not the point.
To register a test with CMake we first need to create an executable:
add_executable(testMaths ut_maths.cpp)
target_link_libraries(testMaths
PRIVATE
Maths
ProjectConfiguration
gtest
gtest_main
)Nothing new here. The gtest and gtest_main targets are declared by the googletest project we’ve included using the FetchContent module. Now that we have an executable target it can be registered using the add_test command:
add_test(NAME testMaths COMMAND testMaths)It’s a simple one-liner, but can actually be a little confusing at first – testMaths everywhere! Let’s unpack what’s going on here.
First an executable target testMaths is declared – testMaths is both the name of the target and the executable binary file itself. Then add_test is called. The NAME argument is the name of the test – this is the name that can later be used to execute this specific test using ctest. The COMMAND argument is the exact command ctest will execute when asked to run the test, this can be an arbitrary command runnable in your shell, but if a name of a CMake target is specified, it will automatically be expanded to the absolute path to the binary produced by that target. So here, the following add_test call would be equivalent to the previous one:
add_test(NAME testMaths COMMAND ${CMAKE_CURRENT_BINARY_DIR}/testMaths)We might as well save a few keystrokes and use the short form.
Now that we have some tests in place and registered for execution let’s actually run them. Make sure to rebuild the project first and then simply run the ctest command in the build directory
$ cmake --build build
$ cd build
$ ctest
Test project /home/user/cmake_fundamentals/build
Start 1: testMaths
1/1 Test #1: testMaths ........................ Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.01 secDoing so, executes all tests previously registered using the add_test command. As you can see the output isn’t much. It just reports if the test passed and how long it took. The output can be made more verbose by specifying the -V flag to ctest
$ ctest -V
...
Test project /home/user/cmake_fundamentals/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: testMaths
1: Test command: /home/user/cmake_fundamentals/build/tests/testMaths
1: Test timeout computed to be: 1500
1: Running main() from /home/user/cmake_fundamentals/build/_deps/googletest-src/googletest/src/gtest_main.cc
1: [==========] Running 2 tests from 1 test suite.
1: [----------] Global test environment set-up.
1: [----------] 2 tests from maths
1: [ RUN ] maths.addTest
1: [ OK ] maths.addTest (0 ms)
1: [ RUN ] maths.subTest
1: [ OK ] maths.subTest (0 ms)
1: [----------] 2 tests from maths (0 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 2 tests from 1 test suite ran. (0 ms total)
1: [ PASSED ] 2 tests.
1/1 Test #1: testMaths ........................ Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.01 secQuite a bit more verbose actually. But we also get the expected output from gtest.
A good alternative to using the -V flag is to use –output-on-failure instead. It does exactly what it says – full output related to a given test will be printed only if it fails.
I have mentioned that using googletest provides some additional convenience for free. Let’s now look at exactly what I meant. This will also give us an opportunity to look at some more ctest parameters useful in day to day work.
gtest_discover_tests
As mentioned CMake provides some additiona support for tests written using googletest. The extra commands are available in the GoogleTest module. Once included a gtest_find_tests and gtest_discover_tests commands can be used. The first one is an older implementation of the same functionality, I’d recommend just using the second one, as it’s simply more reliable. This command essentially replaces add_test for tests written using googletest:
# add_test(NAME testMaths COMMAND testMaths)
include(GoogleTest)
gtest_discover_tests(testMaths)As you can see its use is very simple. Rebuilding the project and running ctest now would produce the following output
$ ctest
Test project /home/user/cmake_fundamentals/build
Start 1: maths.addTest
1/2 Test #1: maths.addTest .................... Passed 0.00 sec
Start 2: maths.subTest
2/2 Test #2: maths.subTest .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.01 secAs you can see ctest now recognizes both test cases contained in a single gtest executable as separate tests. Without going into too much detail the gtest_discover_tests command automatically registers separate tests for each test case defined in the test suite.
CTest offers functionality for executing only tests that match a given regular expression. This is done using the -R argument:
$ ctest -R add
Test project /home/user/cmake_fundamentals/build
Start 1: maths.addTest
1/1 Test #1: maths.addTest .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.00 secThe reverse is also available – tests can be filtered out based on a regular expression. This is done using the -E argument:
$ ctest -E add
Test project /home/user/cmake_fundamentals/build
Start 1: maths.subTest
1/1 Test #1: maths.subTest .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.00 secUsing gtest_discover_tests can be very convenient, however, it can have some drawbacks. In larger projects, with the number of test cases in the hundreds or thousands, this can have a significant runtime cost. Every test case is now executed as a separate process, and the cost of spinning up and tearing down all of these processes can quickly add up and significantly increase the total test execution time. Just something to keep in mind. Most of the time I’d recommend using gtest_discover_tests, since it provides finer granularity.
Summary
In this post I demonstrated how to add and execute tests using facilities provided by CMake – the add_test command and ctest test driver. In preparation of the test environment I have also demonstrated the basic use of the FetchContent module – a very useful tool for managing external dependencies build alongside our project. The general subject of dependency management was also discussed briefly. This is a large, complex topic that deserves a post (or a series) of its own.
0 Comments