Tensorflow: Why must `saver = tf.train.Saver()` be declared after variables are declared?

Question

Important clarification: I was only running this section, the graph definition, in a notebook enviroment. I had not run an actual session yet.

When running this code:

with graph.as_default(): #took out " , tf.device('/cpu:0')"

  saver = tf.train.Saver()
  valid_examples = np.array(random.sample(range(1, valid_window), valid_size)) #put inside graph to get new words each time

  train_dataset = tf.placeholder(tf.int32, shape=[batch_size, cbow_window*2 ])
  train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1])
  valid_dataset = tf.constant(valid_examples, dtype=tf.int32)
  valid_datasetSM = tf.constant(valid_examples, dtype=tf.int32)

  embeddings = tf.get_variable( 'embeddings', 
    initializer= tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))

  softmax_weights = tf.get_variable( 'softmax_weights',
    initializer= tf.truncated_normal([vocabulary_size, embedding_size],
                         stddev=1.0 / math.sqrt(embedding_size)))

  softmax_biases = tf.get_variable('softmax_biases', 
    initializer= tf.zeros([vocabulary_size]),  trainable=False )

  embed = tf.nn.embedding_lookup(embeddings, train_dataset) #train data set is

  embed_reshaped = tf.reshape( embed, [batch_size*cbow_window*2, embedding_size] )


  segments= np.arange(batch_size).repeat(cbow_window*2)

  averaged_embeds = tf.segment_mean(embed_reshaped, segments, name=None)

    #return tf.reduce_mean( tf.nn.sampled_softmax_loss(weights=softmax_weights, biases=softmax_biases, inputs=averaged_embeds,
                               #labels=train_labels, num_sampled=num_sampled, num_classes=vocabulary_size))

  loss = tf.reduce_mean(
    tf.nn.sampled_softmax_loss(weights=softmax_weights, biases=softmax_biases, inputs=averaged_embeds,
                               labels=train_labels, num_sampled=num_sampled, num_classes=vocabulary_size))

  norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keepdims=True))
  normSM = tf.sqrt(tf.reduce_sum(tf.square(softmax_weights), 1, keepdims=True))

  normalized_embeddings = embeddings / norm
  normalized_embeddingsSM = softmax_weights / normSM

  valid_embeddings = tf.nn.embedding_lookup(
    normalized_embeddings, valid_dataset)
  valid_embeddingsSM = tf.nn.embedding_lookup(
    normalized_embeddingsSM, valid_datasetSM)

  similarity = tf.matmul(valid_embeddings, tf.transpose(normalized_embeddings))
  similaritySM = tf.matmul(valid_embeddingsSM, tf.transpose(normalized_embeddingsSM))

I got this error

ValueError: No variables to save

while pointing to this line

saver = tf.train.Saver()

I searched stack overflow and found this answer

Tensorflow ValueError: No variables to save from

So I simply put that line at the bottom of the graph definition like so

with graph.as_default(): #took out " , tf.device('/cpu:0')"

  valid_examples = np.array(random.sample(range(1, valid_window), valid_size)) #put inside graph to get new words each time

  train_dataset = tf.placeholder(tf.int32, shape=[batch_size, cbow_window*2 ])
  train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1])
  valid_dataset = tf.constant(valid_examples, dtype=tf.int32)
  valid_datasetSM = tf.constant(valid_examples, dtype=tf.int32)

  embeddings = tf.get_variable( 'embeddings', 
    initializer= tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
  softmax_weights = tf.get_variable( 'softmax_weights',
    initializer= tf.truncated_normal([vocabulary_size, embedding_size],
                         stddev=1.0 / math.sqrt(embedding_size)))

  softmax_biases = tf.get_variable('softmax_biases', 
    initializer= tf.zeros([vocabulary_size]),  trainable=False )

  embed = tf.nn.embedding_lookup(embeddings, train_dataset) #train data set is
  embed_reshaped = tf.reshape( embed, [batch_size*cbow_window*2, embedding_size] )

  segments= np.arange(batch_size).repeat(cbow_window*2)

  averaged_embeds = tf.segment_mean(embed_reshaped, segments, name=None)

  loss = tf.reduce_mean(
    tf.nn.sampled_softmax_loss(weights=softmax_weights, biases=softmax_biases, inputs=averaged_embeds,
                               labels=train_labels, num_sampled=num_sampled, num_classes=vocabulary_size))

  norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keepdims=True))
  normSM = tf.sqrt(tf.reduce_sum(tf.square(softmax_weights), 1, keepdims=True))

  normalized_embeddings = embeddings / norm
  normalized_embeddingsSM = softmax_weights / normSM

  valid_embeddings = tf.nn.embedding_lookup(
    normalized_embeddings, valid_dataset)
  valid_embeddingsSM = tf.nn.embedding_lookup(
    normalized_embeddingsSM, valid_datasetSM)

  similarity = tf.matmul(valid_embeddings, tf.transpose(normalized_embeddings))
  similaritySM = tf.matmul(valid_embeddingsSM, tf.transpose(normalized_embeddingsSM))

  saver = tf.train.Saver()

And then there were no errors!

Why is this so? The graph definition is only defining the graph, not running anything. Perhaps it's an bug prevention measure?

Answer 1

It does not have to. tf.train.Saver has a defer_build argument that, if set to True, allows you to define variables after it has been constructed. You then need to call build explicitly though.

saver = tf.train.Saver(defer_build=True)
# construct your graph, create variables...
...
saver.build()
graph.finalize()
# go on with training

Answer 2

From the documentation on tf.train.Saver the __init__ method has a parameter var_list with the description:

var_list: A list of Variable/SaveableObject, or a dictionary mapping names 
to SaveableObjects. If None, defaults to the list of all saveable objects.

This suggests that the saver makes a list of variables to save when it's first created, which by default contains all of the variables it can find. If no variables have been made, an error makes sense since there are no variables to save.

Random examples:

import tensorflow as tf
saver = tf.train.Saver()

The above throws an error, and so does below

import tensorflow as tf
x = tf.placeholder(dtype=tf.float32)
saver = tf.train.Saver()

But this last example runs,

import tensorflow as tf
x = tf.Variable(0.0)
saver = tf.train.Saver()