How to properly apply dropout with train_and_evaluate function in tf.contrib.learn.Experiment

Question

I'm using tensorflow high level api tf.contrib.learn.Experiment to run my model. I applied tf.nn.dropout in the model code and use train_and_evaluate function to train the model. However, I can't figure out know how to set argument keep_prob as 1 in tf.nn.dropout only at evaluation within train_and_evaluate process(because generally dropout should be used only at training).

Answer 1

You can use the tf.estimator.ModeKeys to detect whether you are calling your estimator in TRAIN mode or in EVAL mode.

When constructing your model function for your Estimator (as depicted in this tutorial), the function have to respect a skeleton :

def model_fn(features, labels, mode, params):

The mode parameter is a tf.estimator.ModeKeys, so, in your model function, you can simply test against the mode parameter to detect whether you are in training or in evaluation. (or in prediction).

A simple example :

def model_fn(features, labels, mode, params):
    keep_prob = 1.0
    if mode == tf.estimator.ModeKeys.TRAIN:
        keep_prob = 0.8
    [...]

On the side :

Instead of using tf.nn.dropout, consider using tf.layers.dropout. It's a wrapper over tf.nn.dropout, but it comes with a boolean to activate or deactivate the dropout. (You can set the boolean the same way that the prob in my example).