How to detect failing unit test using rake

Question

I am using Ruby 1.9.3 and Rails 3.2.8. This uses rake 0.9.2.2.

I wish to detect a failing unit test and, in Rails 2.3, we did this by checking the exit code. This no longer works.

I try:

rake test:units TEST=test/unit/failing_test.rb RAILS_ENV=test && echo "OK"

failing_test.rb consists of a single test which asserts false. I expect to see a test failure and not to see "OK", but instead, I see:

Started
F
===============================================================================
Failure: <false> is not true.
test: failing test case should fail. (FailingTest)
test/unit/failing_test.rb:6:in `block (2 levels) in <class:FailingTest>'
shoulda-context (1.0.0) lib/shoulda/context/context.rb:398:in `call'
shoulda-context (1.0.0) lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
activesupport (3.2.8) lib/active_support/testing/setup_and_teardown.rb:72:in `block in run'
activesupport (3.2.8) lib/active_support/callbacks.rb:425:in `_run__3774233495015181374__setup__985181848351195933__callbacks'
activesupport (3.2.8) lib/active_support/callbacks.rb:405:in `__run_callback'
activesupport (3.2.8) lib/active_support/callbacks.rb:385:in `_run_setup_callbacks'
activesupport (3.2.8) lib/active_support/callbacks.rb:81:in `run_callbacks'
activesupport (3.2.8) lib/active_support/testing/setup_and_teardown.rb:70:in `run'
===============================================================================


Finished in 0.002753 seconds.

1 tests, 1 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
0% passed

363.24 tests/s, 363.24 assertions/s
rake aborted!
Command failed with status (1): [/Users/chris/.rvm/rubies/ruby-1.9.3-p194/b...]

Tasks: TOP => test:units
(See full trace by running task with --trace)
OK

Specifically, I see "Command failed with status (1)", but this seems to get eaten up by rake as I also see "OK". Note that if I do echo $?, I see the exit code is 0 (success). I'm trying to get this to work in the context of our continuous integration and also with our release script, both of which assume the exit code will be non-zero in the case of an error, as happened with Rails 2.3.

Answer 1

My failing test with the same rails/ruby/rake is exiting 1 unlike yours. Perhaps you have something calling at_exit { exit 0 } or trap('EXIT') { exit 0 } somewhere in your code base. This would modify the exit code for rake.

If you want a non-Ruby way of detecting failures you could try reading the output of the test using grep. grep will return 0 if it finds a match and 1 if it did not.

the regular expression looks for " 0 failures, 0 errors", returning 0 if it found and 1 if not.

rake test:units TEST=test/unit/failing_test.rb RAILS_ENV=test | \
  tee >(xargs -0 echo {}) | \ 
  grep '\([[:space:]]0[[:space:]]\)failures,\1errors' && 
  echo OK