Skipping a CTest if the target was not built

Question

In CMake, one can use add_subdirectory(foo EXCLUDE_FROM_ALL) to exclude the targets defined under foo from being built by default unless they are needed by some other target, like this:

add_subdirectory(foo EXCLUDE_FROM_ALL)
add_custom_target(wanted_foo_targets ALL DEPENDS
  bar_tool  # only build bar_tool from foo
)

Suppose foo defines two targets, bar_tool and foo_tool, and tests for each:

add_test(NAME test_foo_tool COMMAND foo_tool) 
add_test(NAME test_bar_tool COMMAND bar_tool) 

When you build the overall project, foo_tool is not built, because it is not a dependency of anything, but bar_tool is built because it is a dependency of wanted_foo_targets.

But CTest still registers the test_foo_tool tool, which now fails because foo_tool doesn't exist.

Can I remove/hide/skip tests that refer to targets that weren't built? I know I could make the test command something like [ -f foo_tool ] && foo_tool but this will erroneously state that the test passed.

Bonus points if I can do this from the parent directory, without modifying how the tests are set up inside foo. In my case, foo is a project provided by a third party, and I don't want to have to maintain a patch of their code as they write new tests.

Answer 1

This can be done by using the SKIP_RETURN_CODE test property in combination with a small test driver script (which you have already considered using).

In the source code directory add a test driver script named test_driver.sh with the following contents:

#!/bin/sh
if [ -x "$1" ] ; then exec "$1" ; else exit 127 ; fi

The script checks for existence of a test executable passed as the first argument and exits with error code 127 (aka "command not found") if it does not exist.

In your CMakeLists.txt add the tests in the following way:

add_test(NAME test_foo_tool COMMAND "${CMAKE_SOURCE_DIR}/test_driver.sh" $<TARGET_FILE:foo_tool>)
set_tests_properties(test_foo_tool PROPERTIES SKIP_RETURN_CODE 127)

When you run the tests with the executable foo_tool missing, ctest will report the test_foo_tool as a skipped test but not as a failed test.

$ ctest 
...
    Start 1: test_foo_tool
1/2 Test #1: test_foo_tool ....................***Skipped   0.01 sec
    Start 2: test_bar_tool
2/2 Test #2: test_bar_tool ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.03 sec

The following tests did not run:
      1 - test_foo_tool (Skipped)

Answer 2

Tests can be disabled by setting the DISABLED property of the test in question. You can also get a list of tests defined in a sub-directory by checking the TESTS property.

So you should be able to use get_property to get a lists of all the tests for that specific directory and then disable all of them. If only a sub-set of the tests needs to be disabled then filter the list of all tests, using list(FILTER) to just those that need to be disabled. This is only convenient if there is a naming convention that is followed.

Unfortunately set_property can only modify the property of a test within the same directory not a sub-directory. This means that the CMakeLists.txt file would need be updated.

Alternatively if modifying the CMakeLists.txt is too much of an issue just don't run those tests. That could be done by using the ctest -E regex option to exclude those tests.