Menu
I'm currently trying to compare some RNNs and I have an issue only with the LSTM and I have no idea why.
I'm training with the same code/dataset a LSTM, SimpleRNN and GRU. For all of them the loss decrease normally. But for the LSTM, after a certain point (loss around 0.4), the loss directly falls to 10e-8. If I try to predict an output, I only have Nan.
This is the code :
nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64
model = Sequential()
model.add(LSTM( units=nb_unit,
input_shape=inp_shape,
return_sequences=True,
activation='softmax')) # I just change the cell name
model.compile(loss=loss_,
optimizer=optimizer_,
metrics=[metrics_])
checkpoint = ModelCheckpoint("lstm_simple.h5",
monitor=loss_,
verbose=1,
save_best_only=True,
save_weights_only=False,
mode='auto',
period=1)
early = EarlyStopping( monitor='loss',
min_delta=0,
patience=10,
verbose=1,
mode='auto')
history = model.fit(X_train, y_train,
validation_data=(X_test, y_test),
epochs=nb_epoch,
batch_size=batch_size,
verbose=2,
callbacks = [checkpoint, early])
This the the output of GRU and LSTM with the same input :
Input :
[[[1 0 0 0 0 0 0]
[0 1 0 0 0 0 0]
[0 0 0 1 0 0 0]
[0 0 0 1 0 0 0]
[0 1 0 0 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 1 0 0]
[0 0 0 1 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 1 0 0]
[0 0 0 1 0 0 0]
[0 1 0 0 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 1 0 0]
[0 0 0 1 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 0 1 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]]]
LSTM predicts :
[[[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]]]
GRU predicts :
[[[ 0. 0.54 0. 0. 0.407 0. 0. ]
[ 0. 0.005 0.66 0.314 0. 0. 0.001]
[ 0. 0.001 0.032 0.957 0. 0.004 0. ]
[ 0. 0.628 0. 0. 0. 0.372 0. ]
[ 0. 0.555 0. 0. 0. 0.372 0. ]
[ 0. 0. 0. 0. 0.996 0.319 0. ]
[ 0. 0. 0.167 0.55 0. 0. 0. ]
[ 0. 0.486 0. 0.002 0. 0.51 0. ]
[ 0. 0.001 0. 0. 0.992 0.499 0. ]
[ 0. 0. 0.301 0.55 0. 0. 0. ]
[ 0. 0.396 0.001 0.007 0. 0.592 0. ]
[ 0. 0.689 0. 0. 0. 0.592 0. ]
[ 0. 0.001 0. 0. 0.997 0.592 0. ]
[ 0. 0. 0.37 0.55 0. 0. 0. ]
[ 0. 0.327 0.003 0.025 0. 0.599 0. ]
[ 0. 0.001 0. 0. 0.967 0.599 0.002]
[ 0. 0. 0. 0. 0. 0.002 0.874]
[ 0.004 0.076 0.128 0.337 0.02 0.069 0.378]
[ 0.006 0.379 0.047 0.113 0.029 0.284 0.193]
[ 0.006 0.469 0.001 0.037 0.13 0.295 0.193]]]
For the loss, you can find below the last lines of the fit() history:
Epoch 116/250
Epoch 00116: categorical_crossentropy did not improve
- 2s - loss: 0.3774 - categorical_crossentropy: 0.3774 - val_loss: 0.3945 - val_categorical_crossentropy: 0.3945
Epoch 117/250
Epoch 00117: categorical_crossentropy improved from 0.37673 to 0.08198, saving model to lstm_simple.h5
- 2s - loss: 0.0820 - categorical_crossentropy: 0.0820 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08
Epoch 118/250
Epoch 00118: categorical_crossentropy improved from 0.08198 to 0.00000, saving model to lstm_simple.h5
- 2s - loss: 7.5460e-08 - categorical_crossentropy: 7.5460e-08 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08
Or the evolution of the loss based on Epochs.
I previously tried it without Softmax and with MSE as loss function and I didn't get any error.
If needed, you can find the notebook and script to generate the dataset on Github (https://github.com/Coni63/SO/blob/master/Reber.ipynb).
Many thanks for your support, Regards, Nicolas
The root cause seems to be the Softmax function which vanished. If I stop it before it crashed and display the sum of the softmax for every timestep I have :
LSTM :
[[ 0.112]
[ 0.008]
[ 0.379]
[ 0.04 ]
[ 0.001]
[ 0.104]
[ 0.021]
[ 0. ]
[ 0.104]
[ 0.343]
[ 0.012]
[ 0. ]
[ 0.23 ]
[ 0.13 ]
[ 0.147]
[ 0.145]
[ 0.152]
[ 0.157]
[ 0.163]
[ 0.169]]
GRU :
[[ 0.974]
[ 0.807]
[ 0.719]
[ 1.184]
[ 0.944]
[ 0.999]
[ 1.426]
[ 0.957]
[ 0.999]
[ 1.212]
[ 1.52 ]
[ 0.954]
[ 0.42 ]
[ 0.83 ]
[ 0.903]
[ 0.944]
[ 0.976]
[ 1.005]
[ 1.022]
[ 1.029]]
With a Softmax of 0, the next step will try to divide by 0. Now I have no idea how to fix it.
817
I just post my current solution in case someone else face this issue in the future.
To avoid vanishing, I've added a simple Fully-connected layer with the same output size as the input and it worked properly afterward. This layer allows another "configuration" of the output of LSTM/GRU/SRNN and avoid the output to vanish.
This is the final code :
nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64
model = Sequential()
model.add(LSTM(units=nb_unit,
input_shape=inp_shape,
return_sequences=True)) # LSTG/GRU/SimpleRNN
model.add(Dense(7, activation='softmax')) # New
model.compile(loss=loss_,
optimizer=optimizer_,
metrics=[metrics_])
checkpoint = ModelCheckpoint("lstm_simple.h5",
monitor=loss_,
verbose=1,
save_best_only=True,
save_weights_only=False,
mode='auto',
period=1)
early = EarlyStopping(
monitor='loss',
min_delta=0,
patience=10,
verbose=1,
mode='auto')
I hope this can help someone else :)
197
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With