How should I deal with failing tests for bugs that will not be fixed

Question

I have a complex set of integration tests that uses Perl's WWW::Mechanize to drive a web app and check the results based on specific combinations of data. There are over 20 subroutines that make up the logic of the tests, loop through data, etc. Each test runs several of the test subroutines on a different dataset.

The web app is not perfect, so sometimes bugs cause the tests to fail with very specific combinations of data. But these combinations are rare enough that our team will not bother to fix the bugs for a long time; building many other new features takes priority.

So what should I do with the failing tests? It's just a few tests out of several dozen per combination of data. 1) I can't let it fail because then the whole test suite would fail. 2) If we comment them out, that means we miss out on making that test for all the other datasets. 3) I could add a flag in the specific dataset that fails, and have the test not run if that flag is set, but then I'm passing extra flags all over the place in my test subroutines.

What's the cleanest and easiest way to do this? Or are clean and easy mutually exclusive?

Answer 1

That's what TODO is for.

With a todo block, the tests inside are expected to fail. Test::More will run the tests normally, but print out special flags indicating they are "todo". Test::Harness will interpret failures as being ok. Should anything succeed, it will report it as an unexpected success. You then know the thing you had todo is done and can remove the TODO flag.

The nice part about todo tests, as opposed to simply commenting out a block of tests, is it's like having a programmatic todo list. You know how much work is left to be done, you're aware of what bugs there are, and you'll know immediately when they're fixed.

Once a todo test starts succeeding, simply move it outside the block. When the block is empty, delete it.