Obtain output at each timestep in Keras LSTM

Question

What I want to do is give the output of the LSTM at the current timestep as input to the LSTM in the next timestep. So I want the LSTM to predict the word at at the current time step and give this word as input to the next timestep. Is this can be done then:

How can the input and target data be specified during training i.e. in the model.fit() function?

Answer 1

You cannot do it directly in keras but you could do it using a for loop and stateful network. This works like this (assuming that you have your sentences stored as sequences of integers with size=vocabulary_size:

1. Define a stateful network which accepts one word and returns the following one:

   model = Input(batch_size=(batch_size, 1, 1)) 
   model = Embedding(size_of_vocabulary, output_size, input_length=1)(input)
   model = LSTM(lstm_1st_layer_size, return_sequences=True, stateful=True)(model)
   ....
   model = LSTM(lstm_nth_layer_size, return_sequences=True, stateful=True)(model)
   model = Dense(vocabulary_size, activation="softmax")
   
   model.compile(loss="sparse_categorical_crossentropy", optimizer="rmsprop")
   

2. Assuming that you have a numpy array_of_samples of examples (preceding_word, next_word) you could fit it by:

   model.fit(array_of_samples[:,0], array_of_samples[:,1])
   

3. Now you could try to predict stuff in a following manner:

   sentence = [starting_word]
   for i in range(len_of_sequence - 1):
       sentence.append(model.predict(numpy.array([[sentence[i]]).argmax())
   

Now sentence stores your newly created sentence