Tensorflow softmax_cross_entropy_with_logits asks for unscaled log probabilities

Question

I have noticed that tf.nn.softmax_cross_entropy_with_logits asks for "unscaled log probabilities" for the logits argument. However, nowhere have I seen anyone suggest performing a log operation on their NN predictions before submission to this function. Am I missing something here?

Answer 1

You shouldn't perform a log operation. You shouldn't perform anything, actually :-). What this comment is (arguably poorly) trying to say is that each logit is an unrestricted real number (negative or positive, as big or as small as you want). The softmax cross entropy function will then (conceptually) apply the softmax operation (exponentiate, to turn unrestricted real numbers into positive numbers, and then normalize to make them sum to 1) and compute the cross-entropy.

So, tl;dr, feed the outputs of your last linear layer without any normalization or transfer function to this and you won't be wrong.