Problems with using pytest and tf.test.TestCase simultaneously

Question

I have a simple unit test where I check if I can instantiate my Tensorflow class with slightly different parameters. This seems like a great use case for @pytest.mark.parametrize.

However, I've discovered that parametrize is ignored if I my unit tests are methods of a tf.test.TestCase.

For example, when I run pytest on the following code:

class TestBasicRewardNet(tf.test.TestCase):                                                                                                                          
    @pytest.mark.parametrize("env", ['FrozenLake-v0', 'CartPole-v1',                                                                                               
        'CarRacing-v0', 'LunarLander-v2'])                                                                                                                           
    def test_init_no_crash(self, env):                                                                                                                               
        for i in range(3):                                                                                                                                    
            x = BasicRewardNet(env)  

I get the error TypeError: test_init_no_crash() missing 1 required positional argument: 'env'.

To fix this issue, I tried just getting rid of the class wrapper, but that makes me miss out on some automatic Tensorflow test initialization. In particular, now every BasicRewardNet is built in the same TensorFlow graph, and so I need to do something like add a variable scope to avoid conflicts. Adding in this variable scope seems hacky.

@pytest.mark.parametrize("env", ['FrozenLake-v0', 'CartPole-v1',                                                                                               
     'CarRacing-v0', 'LunarLander-v2'])  
def test_init_no_crash(env):                                                                                                                                         
    for i in range(3):                                                                                                                                               
        with tf.variable_scope(env+str(i)):                                                                                                                          
            x = BasicRewardNet(env)   

I'm wondering if anyone knows a way I can cleanly get the best of both worlds? I'd like to be able to use parametrize and get the automatic Tensorflow initialization of tf.test.TestCase at the same time.

Answer 1

As mentionned in the comments by hoefling, it can be solved using tf.test.TestCase.subTest.

class TestBasicRewardNet(tf.test.TestCase):

    @staticmethod
    def my_sub_test(env):
        for i in range(3):                                                                                                                                               
            with tf.variable_scope(env+str(i)):                                                                                                                          
                x = BasicRewardNet(env)

    def test_init_no_crash(env):
        for env in ['FrozenLake-v0', 'CartPole-v1','CarRacing-v0', 'LunarLander-v2']:
            with self.subTest(env):                                                                                                                                                                                                                    
                 self.my_sub_test(env)

To be able to use the subTest features when running with pytest, you should add pytest-subtests in the requirements, otherwise you won't have them!