How to interpret zigzag training loss?

Question

My training data consists of about ~700 unique samples (this is for a regression problem). The data is not shuffled, so the first N samples have the same label (say, the value 1.25), then the next M samples have a the same label (say, 2.99), etc. In total there's around 15 unique labels.

I'm using a simple CNN, as the input is an image (64x64x3). Even with no dropout or any other form of regularization, I can't get the training loss to stabilize close to zero.

What is this pattern of the learning loss an indication of? (gray line is the training loss, orange line is the validation loss).

Answer 1

The only indication you can get from such pattern is that the learning rate is too large, you should decrease it until the loss starts to decrease.