How to inject data into a graph when using an input pipeline?

Question

I am using an initializable iterator in my code. The iterator returns batches of size 100 from a csv dataset that has 20.000 entries. During training, however, I came across a problem. Consider this piece of code:

def get_dataset_iterator(batch_size): 

    # parametrized with batch_size
    dataset = ... 
    return dataset.make_initializable_iterator()


## build a model and train it (x is the input of my model)
iterator = get_dataset_iterator(100)
x = iterator.get_next()
y = model(x)

## L1 norm as loss, this works because the model is an autoencoder
loss = tf.abs(x - y)

## training operator
train_op = tf.train.AdamOptimizer(0.01).minimize(loss)

with tf.Session() as sess:

    for epoch in range(100):

        sess.run(iterator.initializer)

        # iterate through the whole dataset once during the epoch and 
        # do 200 mini batch updates

        for _ in range(number_of_samples // batch_size):
            sess.run(train_op)

        print(f'Epoch {epoch} training done!')

        # TODO: print loss after epoch here

I am interested in the training loss AFTER finishing the epoch. It makes most sense to me that I calculate the average loss over the whole training set (e.g. feeding all 20.000 samples through the network and averaging their loss). I could reuse the dataset iterator here with a batch size of 20.000, but I have declared x as the input.

So the questions are:

1.) Does the loss calculation over all 20.000 examples make sense? I have seen some people do the calculation with just a mini-batch (the last batch of the epoch).

2.) How can I calculate the loss over the whole training set with an input pipeline? I have to inject all of training data somehow, so that I can run sess.run(loss) without calculating it over only 100 samples (because x is declared as input).

EDIT FOR CLARIFICATION:

If I wrote my training loop the following way, there would be some things that bother me:

with tf.Session() as sess:

    for epoch in range(100):

        sess.run(iterator.initializer)

        # iterate through the whole dataset once during the epoch and 
        # do 200 mini batch updates

        for _ in range(number_of_samples // batch_size):
            _, current_loss = sess.run([train_op, loss])

        print(f'Epoch {epoch} training done!')
        print(current_loss)

Firstly, loss would still be evaluated before doing the last weight update. That means whatever comes out is not the latest value. Secondly, I would not be able to access current_loss after exiting the for loop so I would not be able to print it.

Answer 1

1) Loss calculation over the whole training set (before updating weights) does make sense and is called batch gradient descent (despite using the whole training set and not a mini batch).

However, calculating a loss for your whole dataset before updating weights is slow (especially with large datasets) and training will take a long time to converge. As a result, using a mini batch of data to calculate loss and update weights is what is normally done instead. Although using a mini batch will produce a noisy estimate of the loss it is actually good enough estimate to train networks with enough training iterations.

EDIT:

I agree that the loss value you print will not be the latest loss with the latest updated weights. Probably for most cases it really doesn't make much different or change results so people just go with how you have wrote the code above. However, if you really want to obtain the true latest loss value after you have done training (to print out) then you will just have to run the loss op again after you have done a train op e.g.:

for _ in range(number_of_samples // batch_size):
            sess.run([train_op])
            current_loss = sess.run([loss])

This will get your true latest value. Of course this wont be on the whole dataset and will be just for a minibatch of 100. Again the value is likely a good enough estimate but if you wish to calculate exact loss for whole dataset you will have to run through your entire set e.g. another loop and then average the loss:

    ...
    # Train loop
    for _ in range(number_of_samples // batch_size):
        _, current_loss = sess.run([train_op, loss])

    print(f'Epoch {epoch} training done!')

    # Calculate loss of whole train set after training an epoch.
    sess.run(iterator.initializer)
    current_loss_list = []
    for _ in range(number_of_samples // batch_size):
            _, current_loss = sess.run([loss])
            current_loss_list.append(current_loss)
    train_loss_whole_dataset = np.mean(current_loss_list)
    print(train_loss_whole_dataset)

EDIT 2: As pointed out doing the serial calls to train_op then loss will call the iterator twice and so things might not work out nicely (e.g. run out of data). Therefore my 2nd bit of code will be better to use.