Best practice: testing code which uses a random number generator

Question

Say somewhere (deep, deep down) in the code I am using random numbers to do something¹ cool. Tests should in general be deterministic, while the behavior of the release version of this part of the code explicitly makes use of the non-determinism/randomness of the random number generator.

So to get deterministic (functional) tests, I would like to set a fixed seed value:

size_t seed = 42;
std::mt19937 rng;
rng.seed(seed);

But I also want to make sure that nothing strange (i.e., an exception) happens when I use random input and a random seed:

std::mt19937 rng;
rng.seed(std::random_device()());

Obviously, a finite number of test runs can not ascertain that the code is correct; however, a large number of test runs can at least give you some confidence.

How do I best handle this? I thought about adding something like:

size_t seed = std::random_device()();
#ifdef TESTING
    seed = 42;
#endif
rng.seed(seed);

but then, I can not have one test file (using gtest) that uses a random seed for some tests and a constant seed for some other tests (can I?).

¹ In my case: I choose one element from n uniformly and independently at random to partition the input.

EDIT: I am asking about functional tests, not unit tests.

Answer 1

It depends what kind of test you want to do.

In unit tests, the seed should be set to a fixed value.

In some kind of functional test, where you really want to test the behavior using random numbers, and where it doesn't matter how long it takes, you could set random seed (maybe using time()), and execute the test as many times as you want.

This means, you should create two sets of tests, in similar way (you use gtest), where unit tests would execute much faster. With unit tests, you do not access files and slow resources (like network and databases).

Answer 2

First of all, create an interface for the random number generator, such that test will be able to replace it with the pre-defined one. If you use a random number generator in your unit test it is no longer unit.

Then think about what do you really want to test. I guess, that this algorithm should be tested:

• Whether it works at all (your predefined "random" generator may return 1, 2, 3, ... at each iteration - possibly modulo the maximum range);
• Whether it returns valid results if random generator always return border values (0 and max-1 or max, depending on your needs);
• Whether it notifies an error if random generator returns value outside the range.

---

Edit: (in response to question edit)

Why do you bother about the random number generator in functional tests? Functional tests check, whether application works correctly when used by user. User won't have access to seed of random number generator, so just leave it as it is and allow it to work as you designed it. If the outcome of your algorithm doesn't depend on the values generated by the random number generator, check if they doesn't change between tests. If they do depend on the random number generator, check if they change between two tests (if they don't, that means, the code doesn't do what it is supposed to).