Unit testing with random data

Question

I've read that generating random data in unit tests is generally a bad idea (and I do understand why), but testing on random data and then constructing a fixed unit test case from random tests which uncovered bugs seems nice. However I don't understand how to organize it nicely. My question is not related to a specific programming language or to a specific unit test framework actually, so I'll use python and some pseudo unit test framework. Here's how I see coding it:

def random_test_cases():
   datasets = [
       dataset1,
       dataset2,
       ...
       datasetn
   ]
   for dataset in datasets:
       assertTrue(...)
       assertEquals(...)
       assertRaises(...)
       # and so on

The problem is: when this test case fails I can't figure out which dataset caused failure. I see two ways of solving it:

1. Create a single test case per dataset — the problem is load of test cases and code duplication.
2. Usually test framework lets us pass a message to assert functions (in my example I could do something like assertTrue(..., message = str(dataset))). The problem is that I should pass such a message to each assert, which does not look like elegant too.

Is there a simpler way of doing it?

Answer 1

I still think it's a bad idea.

Unit tests need to be straightforward. Given the same piece of code and the same unit test, you should be able to run it infinitely and never get a different response unless there's an external factor coming in to play. A goal contrary to this will increase maintenance cost of your automation, which defeats the purpose.

Outside of the maintenance aspect, to me it seems lazy. If you put thought in to your functionality and understand the positive as well as the negative test cases, developing unit tests are straightforward.

I also disagree with the user who shows how to do multiple tests cases inside of the same test case. When a test fails, you should be able to tell immediately which test failed and know why it failed. Tests should be as simple as you can make them and as concise/relevant to the code under test as possible.

Answer 2

You could define tests by extension instead of enumeration, or you could call multiple test cases from a single case.

calling multiple test cases from a single test case:

MyTest()
{
    MyTest(1, "A")
    MyTest(1, "B")
    MyTest(2, "A")
    MyTest(2, "B")
    MyTest(3, "A")
    MyTest(3, "B")
}

And there are sometimes elegant ways to achieve this with some testing frameworks. Here is how to do it in NUnit:

[Test, Combinatorial]
public void MyTest(
    [Values(1,2,3)] int x,
    [Values("A","B")] string s)
{
    ...
}