Build Events
A first attempt one might make when dealing with a similar problem is to try to utilize the following form of add_custom_command:
add_custom_command(TARGET <target>
PRE_BUILD | PRE_LINK | POST_BUILD
COMMAND command1 [ARGS...]
...
)This would result in the specified command1 to be executed with the given ARGS… just before the <target> is built (PRE_BUILD) or linked (PRE_LINK) or just after it is built (POST_BUILD). Seems quite straightforward. To illustrate the point let’s assume that we have the following items we need to handle:
Foo – builds an executable.
config.json – configuration file that Foo depends on, and expects to be located in the same directory.
Given the above we might want to copy config.json to the location of executable Foo. This could be done as follows:
add_custom_command(TARGET Foo
POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different
config.json # config file, relative to current directory
$<TARGET_DIRECTORY:Foo> # destination directory, expands to directory of executable Foo
)Building the project with this command in place would result in config.json to be copied to the directory of executable foo, as expected. However, if you make some changes to config.json, without making any other edits, that would cause the target Foo to be recompiled, the changes to config.json would not be propagated to the copy located at the directory of Foo. This is explicitly stated in the add_custom_command documentation:
"The command becomes part of the target and will only execute when the target itself is built."In other words, the build system is in no way aware of the config.json file, and does not follow the changes made to it – it’s a passive operand of the command. Only if the target Foo requires processing, will the custom command be executed. This is fine most of the time, however, you may require a setup where you’d like the custom command to be executed independently of the Foo target.
Luckily add_custom_command has another form, that addresses this issue.
Making the build system aware of arbitrary resources
The main form of the add_custom_command adds an arbitrary build rule to the build system, while also giving us an opportunity to make the build system aware of any additional files we’d like to manage as well as the files produced by executing the command. Once the build system is aware of what files it needs to keep track of, it’s able to correctly deduce the targets and commands that need to be processed after any edits are made. The (slightly simplified) form of the add_custom_command looks as follows:
add_custom_command(OUTPUT output1 [output2 ...] # list of files being the result of executing the command
COMMAND command1 [ARGS] [args1...] # one or more commands to execute
[COMMAND command2 [ARGS] [args2...] ...]
[DEPENDS [depends...]] # dependencies of the command - i.e. the files that we'd like
# the build system to keep track of
[APPEND] # request to add the given command to the set of commands
# that produce the given OUTPUT(s)
)This form of the command may be a little difficult to understand at first, and the documentation may not feel that helpful. However, it’s actually relatively straightforward once you understand the reasoning behind it.
A custom command of this form will execute the given commands, and will result in the files specified in OUTPUT being generated – these are assumed to be relative to the current binary directory, unless absolute paths are specified. The given DEPENDS are the additional resources we’d like to make the build system aware of – touching any files specified here, will result in the command being executed again. The APPEND flag might be the most puzzling one. It allows us to specify the list of commands piece-meal – one-by-one, for the given set of OUTPUT(s). This makes it possible to easily add commands with a foreach loop.
Let’s see how we could take advantage of this form to manage our config.json file:
add_custom_command(OUTPUT config.json
COMMAND ${CMAKE_COMMAND} -E copy_if_different
${CMAKE_CURRENT_SOURCE_DIR}/config.json
$<TARGET_FILE_DIR:Foo>
DEPENDS config.json
)The above adds a custom command that copies the config.json file from the source tree into the build tree. More specifically into the directory that will contain the Foo executable. However, by itself, this custom command does now achieve anything, it needs to be attached to a target, so that the build system knows when to execute it. We could attach it to the Foo target itself, unfortunately the command already implicitly depends on that target, due to the TARGET_FILE_DIR generator expression. We have a choice of either abandoning the generator-expression approach and specifying the target path directly, or alternatively we could attach the command to a different target, independent of Foo. Let’s go with the latter approach for now.
add_custom_target(copyConfig ALL DEPENDS $<TARGET_FILE_DIR:main>/config.json)The copyConfig target is part of the make-all build rule (i.e. it’s always executed if necessary) and it depends on the config.json file existing in the directory of executable Foo.
Running the build system now would result in the config.json being copied into the expected directory. Moreso, making changes to config.json, without touching any other files and rerunning the build would copy the update file. This may seem like a silly contrived example but think back to the case of managing shader files, or other assets mentioned at the start of this post. Or consider a use case of automatically populating a staging area for testing in a cross-compiling environment, and this feature will start looking more and more useful.
Generalizing the implementation
You may have noticed that the amount of code required for such an apparently simple operation is quite large. This is because we’re doing much more than just simply copying a file – making the build system aware of all of the dependencies is what requires all this work. We’d like to enjoy the benefits of doing so, without being so explicit for each file that needs to be handled in a similar manner. This could be done with a simple foreach loop in conjunction with the APPEND flag.
Let’s assume we have a number of resources we’d like to manage:
res1.json
res2.json
res3.jsonFirst the entire OUTPUT set of the custom command needs to be declared:
set(resources res1.json res2.json res3.json)
add_custom_command(OUTPUT ${resources})Once the OUTPUT set is declared we can associate an arbitrary number of commands with it using the APPEND flag:
foreach(resource IN LISTS resources)
add_custom_command(OUTPUT ${resources}
COMMAND ${CMAKE_COMMAND} -E copy_if_different
${CMAKE_CURRENT_SOURCE_DIR}/${resource}
$<TARGET_FILE_DIR:Foo>
APPEND
)
endforeach()Note that each invocation of the add_custom_command needs to specify the exact same OUTPUT set. Specifying a different output list declares another, independent custom command. The APPEND flag then associates the given command with that OUTPUT set, appending it to the list of commands that will be executed to generate it.
Once all of the commands are added, a target needs to be introduced, just like before:
list(TRANSFORM resources PREPEND "$<TARGET_FILE_DIR:Foo>/")
add_custom_target(copyResources ALL DEPENDS ${resources})Using these building blocks we should be able to create arbitrarily complex wrappers to accommodate our use cases. With a bit more work many of the complexities exposed in the above examples could be hidden away, leading to something like the following:
CopyIfDifferent(CopyResources
OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
FILES
config.json res1.json res2.json res3.json
)
add_dependencies(Foo CopyResources)The above assumes some different choices – OUTPUT_DIRECTORY is declared explicitly, so that the command can be associated explicitly with the target that depends on it.
A full implementation may be found in a gist: CopyIfDifferent. The code is no more complex than in this post and adequately commented, so it should be understandable.
Summary
Making the build system aware of all the resources your project depends on may seem like a lot of work, however, I’d argue it’s worth the effort. Everyone involved in the project will quickly come to appreciate the well-managed dependencies and lack of issues with the build system. Project documentation often mentions things like “configure and build the project at least twice (…)”, or “after doing X or Y a clean rebuild is required”. You should be able to avoid most, if not all, of such problems by making all the resources you depend on an explicit part of the build system. The add_custom_command and add_custom_targets are major tools in helping you do so.
Is there a way to not copy these resources on Windows, but rather refer them in build? (if the resources (config1.json, 2.json, … ) are too large in size)
symlinks don’t seem to work in windows.