CMake Fundamentals Part 8

In the previous part of the series, I discussed how to handle project installation. I did however skip the topic of shared libraries. Rather than postponing it any further, I will present the missing information in this post. In the usual fashion let’s try to take the quickest path to our goal and taclke any issues as they show up.

Building shared libraries

Just a quick reminder of where we left of in the previous post. We ended up with a two-project setup, where the cmake_fundamentals project would build only the Maths library and a test_project would use an installed Maths library in a trivial executable. The setup looks more or less as follows:

.
├── cmake_fundamentals
│   ├── Add
│   ├── Subtract
│   ├── Maths
│   ├── tests
│   └── CMakeLists.txt
└── test_project
    ├── main.cpp
    └── CMakeLists.txt

As a refresher let’s configure and build the projects and run the test suite. Here are the commands:

$ cmake -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_SHARED_LIBS:BOOL=OFF -DCMAKE_INSTALL_PREFIX:PATH=$(pwd)/install -B cmake_fundamentals/build -S cmake_fundamentals
$ cmake --build cmake_fundamentals/build -j8 --target install
$ cmake --build code/build -j4 --target install
[16/17] Install the project...
-- Install configuration: "Release"
(...)
-- Installing: /home/user/install/lib/libMaths.a
(...)
-- Installing: /home/user/install/lib/cmake/Maths/Maths.cmake
-- Installing: /home/user/install/lib/cmake/Maths/Maths-debug.cmake
-- Installing: /home/user/install/lib/cmake/Maths/MathsConfig.cmake
-- Installing: /home/user/install/lib/cmake/Maths/MathsConfigVersion.cmake

The tests can be executed from the cmake_fundamentals/build directory by simply running ctest:

$ ctest
Test project /home/user/cmake_fundamentals/build
    Start 1: maths.addTest
1/2 Test #1: maths.addTest ....................   Passed    0.00 sec
    Start 2: maths.subTest
2/2 Test #2: maths.subTest ....................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

The test_project needs to be provided with context necessary to find the installed Maths library, we do so explicity by supplying the CMAKE_PREFIX_PATH variable with a path to the install directory:

$ cmake -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_PREFIX_PATH:PATH=(pwd)/install -B test_project/build -S test_project
$ cmake --build test_project/build -j4
$ ./test_project/build/MathsDemo
42 + 102 = 144
Error calculating 42 + 2147483647

Right, everything seems to work just fine – the tests pass, the installed library can be successfully linked into an executable. Let’s now see what happens if we try building a shared library instead:

$ rm -rf cmake_fundamentals/build/ test_project/build/ install/
$ cmake -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_INSTALL_PREFIX:PATH=$(pwd)/install -B cmake_fundamentals/build -S cmake_fundamentals
$ cmake --build code/build -j4 --target install
[16/17] Install the project...
-- Install configuration: "Release"
(...)
-- Installing: /home/user/install/lib/libAdd.so
(...)
-- Installing: /home/user/install/lib/libSubtract.so
(...)
-- Installing: /home/user/install/lib/libMaths.so
-- Set runtime path of "/home/user/install/lib/libMaths.so" to ""
(...)
-- Installing: /home/user/install/lib/cmake/Maths/Maths.cmake
-- Installing: /home/user/install/lib/cmake/Maths/Maths-release.cmake
-- Installing: /home/user/install/lib/cmake/Maths/MathsConfig.cmake
-- Installing: /home/user/install/lib/cmake/Maths/MathsConfigVersion.cmake

Some new output about the runtime path shows up, but the project builds and installs just fine. How about running tests?

$ cd cmake_fundamentals/build
$ cd cmake_fundamentals/build
$ ctest
Test project /home/user/cmake_fundamentals/build
    Start 1: maths.addTest
1/2 Test #1: maths.addTest ....................   Passed    0.00 sec
    Start 2: maths.subTest
2/2 Test #2: maths.subTest ....................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

That seems to work fine as well. Let’s check the final piece of the puzzle – building and running an executable with the installed library:

$ cmake -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_PREFIX_PATH:PATH=(pwd)/install -B test_project/build -S test_project
$ cmake --build test_project/build -j4
$ ./test_project/build/MathsDemo
./test_project/build/MathsDemo: error while loading shared libraries: libAdd.so: cannot open shared object file: No such file or directory

Everything works fine up to the point where we try to actually run the executable. Why is that? Also, we were able to run the tests – also an executable – just fine. We could conclude that an issue lies with the installation process itself then. This is more or less the case. I already pointed out an important clue along the way – the runtime path of the shared library.

Let’s take a step back and discuss what it takes to use a shared library.

Using Shared Libraries

Static libraries are quite straightforward. We just need to make sure that the compiler is able to find it and then everything will be compiled and linked into the final executable. Shared libraries are a little different – they’re already binaries in the ELF format, rather than just a bag of objects. This means that both the compiler and the resulting executable need to be able to locate the library. This is because shared libraries are not compiled into the executable, but rather are dynamically loaded at runtime. The compiler only ensures that all the required symbols are defined, but actual loading happens when the executable runs.

I’ll spare you the trouble of running all the commands manually this time and just present the relevant output. Here’s how CMake ensures the compiler has the necessary context:

c++ -isystem /home/user/install/include -g -o CMakeFiles/MathsDemo.dir/main.cpp.o -c /home/user/test_project/main.cpp
c++ -g CMakeFiles/MathsDemo.dir/main.cpp.o -o MathsDemo  -Wl,-rpath,/home/user/install/lib /home/user/install/lib/libMaths.so /home/user/install/lib/libAdd.so -Wl,-rpath-link,/home/user/install/lib

As you can see nothing really has changed in the object file compilation step, however, linking is completely different than before. What is all this -Wl,-rpath about?

The -Wl flag itself informs the compiler that the comma-separated list of flags that follow should be forwarded to the linker. This is compiler-specific – other compilers may use a different prefix then -Wl, but CMake handles this for us, so don’t worry about that.

The -rpath flag informs the linker what paths the dynamic loaded should search when trying to load the library at runtime. The linker does this byt setting a RPATH entry in the executable. The -rpath-link is not strictly required – it is used to check at link-time that all symbols in the final executable can be resolved, also for transitive dependencies (libMaths.so -> libAdd.so). Hold on, so what’s the problem? All the paths are specified, shouldn’t the loading happen correctly? Paths are specified, but only for the MathsDemo executable.

If you have any experience with shared libraries you may be thinking “just set LD_LIBRARY_PATH and be done with it”. That seems to be the common quick-fix solution to problems like these. And it would of course work:

$ LD_LIBRARY_PATH=/home/user/install/lib/ ./test_project/build/MathsDemo
42 + 102 = 144
Error calculating 42 + 2147483647

If you’re unfamiliar with LD_LIBRARY_PATH, it’s an environment variable that can be configured with paths the dynamic linker should search when searching for libraries to load. What’s the problem then? If it works, it works, right? Well, not quite. First of all the problem with LD_LIBRARY_PATH is that’s it’s essentially a global variable – if you just set it in your environment it will be used by all executables (and shared libraries loading their own transitive dependencies!). Also, the installation wouldn’t be modular or relocatable – we’d need to do additional scripting on top of all the work CMake has already done, which is obviously less than ideal.

Instead, let’s investigate what the root cause of the problem is. But first, let’s reiterate some key points (and introduce some new information).

• Executables and shared libraries are both binaries in the ELF format.
• RPATH is an entry in an ELF binary, containing dynamic loading search paths.
• RUNPATH has the same function as RPATH, but it has different priority.
• LD_LIBRARY_PATH is an environment variable containig dynamic loading search paths.

Just to elaborate on the RPATH/RUNPATH right away. The search priority priority is different, the relation is a little convoluted:

• If only RPATH is used it has the highest priority – LD_LIBRARY_PATH can not be used to override the paths.
• If RUNPATH is used RPATH is ignored.
• RUNPATH has lower search priority than LD_LIBRARY_PATH.

The end. The purpose of RUNPATH is to enable LD_LIBRARY_PATH to be used for what it’s actually meant to be used for – temporary/ad-hoc substitution of the dynamically loaded dependencies for testing or debugging purposses. RPATH prevents this use case. Long story short – prefer RUNPATH over RPATH in all new code. This seems to be the default now, so no need to worry.

Now, back to investigating what’s wrong. We know that CMake has configured the RPATH on the MathsDemo executable, to contain the following paths:

• /home/user/install/lib
• /home/user/install/lib/libMaths.so
• /home/user/install/lib/libAdd.so

All of these paths are correct, but trying to run the executable still results in an error:

./test_project/build/MathsDemo: error while loading shared libraries: libAdd.so: cannot open shared object file: No such file or directory

Setting LD_LIBRARY_PATH “fixes” the issue:

$ LD_LIBRARY_PATH=/home/user/install/lib/ ./test_project/build/MathsDemo
42 + 102 = 144
Error calculating 42 + 2147483647

The main clue is that LD_LIBRARY_PATH is used by all executables and shared libraries to do their dynamic loading. Could the problem be a transitive dependency then? But the path to libAdd.so is given explicitly, but only to the MathsDemo executable. A problem lies with the libMaths.so – previously we’ve seen the following message

-- Set runtime path of "/home/user/install/lib/libMaths.so" to ""

Meaning that the RPATH of libMaths.so is empty. CMake does this at installation to prevent hardcoding absolute paths, which would essentially never be correct.

To see exactly what’s going on here we’re going to use two tools:

• readelf – used for analysis of ELF binaries.
• ldd – used for analysis of dynamic dependencies

This will not be a comprehensive guide on how to use these though. Instead I’ll just present the necessary commands.

We can use readelf to print the .dynamic section of an ELF binary – these are the dynamically loaded dependencies required for the binary to work. Let’s inspect MathsDemo and libMaths.so:

$ readelf -d ./test_project/build/MathsDemo | grep -E 'NEEDED|RUNPATH|RPATH'
 0x0000000000000001 (NEEDED)             Shared library: [libMaths.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/home/user/install/lib]

$ readelf -d ./install/lib/libMaths.so | grep -E 'NEEDED|RUNPATH|RPATH'
 0x0000000000000001 (NEEDED)             Shared library: [libAdd.so]
 0x0000000000000001 (NEEDED)             Shared library: [libSubtract_d.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]

Note that MathsDemo requires libMaths.so and has RUNPATH configured, libMaths.so requires libAdd.so, but doesn’t have any RUNPATH information, just as we’d expect, since the runpath was cleared on installation.

Let’s see what ldd will tell us:

$ ldd ./test_project/build/MathsDemo 
        (...)
        libMaths.so => /home/user/install/lib/libMaths.so (0x00007f80f71b9000)
        (...)
        libAdd.so => not found
        libSubtract.so => not found
        (...)

$ ldd ./install/lib/libMaths.so 
        (...)
        libAdd.so => not found
        libSubtract.so => not found
        (...)

Yup, it explicitly confirms what we could deduce from the previous information – the transitive dependencies are not found. The way to fix this correctly is to configure RUNPATH install paths.

Install RUNPATH

If we don’t explicitly configure the installation RUNPATH it will always be cleared, as already mentioned. This is because it’s impossible for CMake to deduce a default that would be always correct – it depends on your specific installation layout. Once we know what the problem is, the fix is actually quite simple. Here are the requirements:

1. libMaths.so needs to be able to find both libAdd.so and libSubtract.so
2. Installation needs to be relocateable

The second point can be satisfied using a feature of modern linux systems, rather than CMake. The RPATH and RUNPATH entries support a special $ORIGIN value, which expands to the directory the library is located at. This makes it easy to configure RUNPATH in tearms relative to the library installation directory. Here’s a reminder of what our installation directory looks like:

.
├── bin/        # executable binaries
├── include/    # header files
└── lib/        # static or shared libraries

This means that we’d like executables located in bin/ to be able to find shared libraries located in lib/, and shared libraries to be able to locate other shared libraries located in lib/ as well. The RUNPATH setting that achieves both, could be defined as follows (in pseudocode):

RUNPATH = $ORIGIN:$ORIGIN/../lib/

In CMake this is done by setting the CMAKE_INSTALL_RPATH cache variable. Rather than hardcoding the relative path between the executables install directory and library install directory (as shown above) we can generalize this a little and compute the relative path using the file command:

file(RELATIVE_PATH relativeRpath
    ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}
    ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}
)
set(CMAKE_INSTALL_RPATH $ORIGIN $ORIGIN/${relativeRpath})

The above needs be configured before any targets are created. Once this is done, re-building will result in libraries with updated runpaths being installed:

$ cmake -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_INSTALL_PREFIX:PATH=$(pwd)/install -B cmake_fundamentals/build -S cmake_fundamentals
$ cmake --build code/build -j4 --target install
[4/5] Install the project...
-- Install configuration: "Debug"
-- Installing: /home/user/install/lib/libAdd.so
-- Set runtime path of "/home/user/install/lib/libAdd.so" to "$ORIGIN:$ORIGIN/../lib"
(...)
-- Installing: /home/user/install/lib/libSubtract.so
-- Set runtime path of "/home/user/install/lib/libSubtract.so" to "$ORIGIN:$ORIGIN/../lib"
(...)
-- Installing: /home/user/install/lib/libMaths.so
-- Set runtime path of "/home/user/install/lib/libMaths.so" to "$ORIGIN:$ORIGIN/../lib"
(...)

Once the test_project is re-built the MathsDemo can be executed successfully without manipulating LD_LIBRARY_PATH.

$ cmake -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_PREFIX_PATH:PATH=(pwd)/install -B test_project/build -S test_project
$ cmake --build test_project/build -j4
$ ./test_project/build/MathsDemo
42 + 102 = 144
Error calculating 42 + 2147483647

And that’s the most important part done right there. There’s two more details to handle – versioning and symbol visibility.

Shared Library Versioning

If you intend to deploy the library you’re developing for use by other projects it’s a good idea to ensure consistent versioning, that correctly advertises compatibility across different versions. For the sake of discussion let’s assume that we’re following the semantic versioning. To match this convention we need to specify that minor and patch version changes guarantee API and ABI compatibility. CMake can handle all of the intricacies of encoding this information into the libraries regardless of the target platform. This is done by setting the VERSION and SOVERSION target properties.

The VERSION property should be set to the full major.minor.patch project version.
The SOVERSION property should be set to the highest version subcomponent that guarantees API/ABI compatibility.

This is done as follows:

set_target_properties(Maths
    PROPERTIES
        VERSION ${PROJECT_VERSION}
        SOVERSION ${PROJECT_VERSION_MINOR}
)

The same needs to be repeated for all installed targets.
Once done, the project can be re-build:

[7/8] Install the project...
-- Install configuration: "Debug"
-- Installing: /home/user/install/lib/libAdd.so.0.1.0
-- Installing: /home/user/install/lib/libAdd.so.1
-- Set runtime path of "/home/user/install/lib/libAdd.so.0.1.0" to "$ORIGIN:$ORIGIN/../lib"
-- Installing: /home/user/install/lib/libAdd.so
(...)
-- Installing: /home/user/install/lib/libSubtract.so.0.1.0
-- Installing: /home/user/install/lib/libSubtract.so.1
-- Set runtime path of "/home/user/install/lib/libSubtract.so.0.1.0" to "$ORIGIN:$ORIGIN/../lib"
-- Installing: /home/user/install/lib/libSubtract.so
(...)
-- Installing: /home/user/install/lib/libMaths.so.0.1.0
-- Installing: /home/user/install/lib/libMaths.so.1
-- Set runtime path of "/home/user/install/lib/libMaths.so.0.1.0" to "$ORIGIN:$ORIGIN/../lib"
-- Installing: /home/user/install/lib/libMaths.so

As you can see not one but three files associated with each shared library are now installed. Only one of these files is an actual binary – the fully-versioned .so, the remaining two are symbolic links.

$ ll install/lib
lrwxrwxrwx libMaths.so -> libMaths.so.1
lrwxrwxrwx libMaths.so.1 -> libMaths.so.0.1.0
-rw-r--r-- libMaths.so.0.1.0

Summary

Developing shared libraries is more difficult than it may seem. I only scratched the surface of how to at least get the library to link correctly both a compile-time and runtime. That’s just the tip of the iceberg, though. The most important part is actually ensuring the API and ABI stability, but this goes well beyond the scope of this series.

I omitted one aspect of defining shared libraries that I had mentioned – symbol visibility. It will be covered in the future in a post outside of the CMake Fundamentals series.

 

References

• Git gist
• ldd
• readelf
• CMAKE_INSTALL_RPATH
• VERSION
• SOVERSION