Tensorflow tf.losses.cosine_distance is greater than one

Question

I'm training a Neural network on Tensorflow and I'm using tf.losses.cosine_distance as a loss function.

The training proceeds well, but my concern is that during the training I have values of losses > 1. The cosine distance (if the input tensors are normalized to 1), should always be a value less than one? How is the loss calculated? Is it a sum of losses in a batch?

Answer 1

Correct, tf.losses.cosine_distance has reduction argument, which equals to reduction=Reduction.SUM_BY_NONZERO_WEIGHTS by default:

cosine_distance(
    labels,
    predictions,
    dim=None,
    weights=1.0,
    scope=None,
    loss_collection=tf.GraphKeys.LOSSES,
    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS      # <-- Here
)

In this case, it computes the sum of all cosine distances across the batch. Change it to Reduction.MEAN and you'll have a mean loss across the batch, which is usually what you want.