Saving checkpoints and resuming training in tensorflow

Question

I was playing with saving checkpoints and resuming training from saved checkpoints. I was following the example given in - https://www.tensorflow.org/versions/r0.8/api_docs/python/train.html#import_meta_graph To keep things simple, I have not used any 'real' training of a network. I just performed a simple subtraction op and each check point saves the same operation on same tensors again and again. A minimal example is provided in the form of the following ipython notebook - https://gist.github.com/dasabir/29b8f84c6e5e817a72ce06584e988f10

In the first phase, I'm running the loop for 100 times (by setting the value of the variable 'endIter = 100' in the code) and saving checkpoints every 10th iteration. So, the checkpoints being saved are numbered as - 9, 19, ..., 99. Now when I'm changing the 'enditer' value to say 200 and am resuming training, the checkpoints again start to be saved from 9, 19, ... (not 109, 119, 129, ...). Is there a trick I'm missing?

Answer 1

Can you print out 'latest_ckpt', and see if it points to the latest ckpt file? Also, you need to maintain the global_step using a tf.variable:

global_step = tf.Variable(0, name='global_step', trainable=False)
...
ckpt = tf.train.get_checkpoint_state(ckpt_dir)
if ckpt and ckpt.model_checkpoint_path:
    print ckpt.model_checkpoint_path
    saver.restore(sess, ckpt.model_checkpoint_path) # restore all variables
start = global_step.eval() # get last global_step
print "Start from:", start

for i in range(start, 100):
...
    global_step.assign(i).eval() # set and update(eval) global_step with index, i
    saver.save(sess, ckpt_dir + "/model.ckpt", global_step=global_step)

You can take a look at the full example:

https://github.com/nlintz/TensorFlow-Tutorials/pull/32/files