Valgrind fail does not fail CTest

Question

I'm struggling to add memory check test for my travis build.

Normally I run my tests with ctest --verbose . I get a nice output finished with:

2: [ RUN      ] ContainsNoneTest.GivenSomeOfTheValuesShouldReturnFalse
2: [       OK ] ContainsNoneTest.GivenSomeOfTheValuesShouldReturnFalse (0 ms)
2: [----------] 4 tests from ContainsNoneTest (0 ms total)
2: 
2: [----------] Global test environment tear-down
2: [==========] 49 tests from 13 test cases ran. (1 ms total)
2: [  PASSED  ] 49 tests.
2/2 Test #2: LangTest .........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.20 sec

Now, I want to add valgrind. I put a small leak in one of the methods, run the test with ctest --verbose -T memcheck:

2: [ RUN      ] ContainsNoneTest.GivenSomeOfTheValuesShouldReturnFalse
2: [       OK ] ContainsNoneTest.GivenSomeOfTheValuesShouldReturnFalse (2 ms)
2: [----------] 4 tests from ContainsNoneTest (8 ms total)
2: 
2: [----------] Global test environment tear-down
2: [==========] 49 tests from 13 test cases ran. (453 ms total)
2: [  PASSED  ] 49 tests.
2/2 MemCheck #2: LangTest .........................   Passed    1.00 sec
2: process test output now: LangTest LangTest
PostProcessTest memcheck results for : LangTest

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   6.56 sec
-- Processing memory checking output:
2/2 MemCheck: #2: LangTest .........................   Defects: 12
MemCheck log files can be found here: ( * corresponds to test number)
/home/rumcajs/CLionProjects/ModernCppChallenge/cmake-build-debug/Testing/Temporary/MemoryChecker.*.log
Memory checking results:
Memory Leak - 12

Awesome, valgrind detected the leaks. Unfortunately the process exited with return code 0 (checked with echo $?). Is there a way to change ctest behaviour to output non-zero return code when a leak is detected? Regexing the stdout or valgrind output files seems uncivilized and barbaric and I'd like to avoid it.

Answer 1

Based on the error-exitcode comment above, I've set up the following in my travis scripts.

First, add the error-exitcode option when configuring your CMake project:

cmake -D MEMORYCHECK_COMMAND_OPTIONS="--error-exitcode=1 --leak-check=full" $SOURCE

Then to test, I run:

  if ! ctest -D ExperimentalMemCheck --output-on-failure; then
    find Testing/Temporary -name "MemoryChecker.*.log" -exec cat {} +
    exit 1
  fi

The extra find and cat bit is because --output-on-failure only shows the test output, not the valgrind output.