TensorFlow: LSTM on top of another LSTM

Question

There is not much to say as an introduction: I wanted to stack LSTM on another LSTM in TensorFlow, but keep being stopped by mistake I cannot quite understand, let alone solve singlehandedly.

Here's the code:

def RNN(_X, _istate, _istate_2, _weights, _biases):

    _X = tf.transpose(_X, [1, 0, 2])  
    _X = tf.reshape(_X, [-1, rozmiar_wejscia]) 
    _X = tf.matmul(_X, _weights['hidden']) + _biases['hidden']

    lstm_cell = rnn_cell.BasicLSTMCell(ukryta_warstwa, forget_bias=1.0)
    _X = tf.split(0, liczba_krokow, _X)

    outputs, states = rnn.rnn(lstm_cell, _X, initial_state=_istate)


    lstm_cell_2 = rnn_cell.BasicLSTMCell(ukryta_warstwa, forget_bias = 1.0)
    outputs2, states2 = rnn.rnn(lstm_cell_2, outputs, initial_state = _istate_2)

    return tf.matmul(outputs2[-1], _weights['out']) + _biases['out']

And what I keep receiving is:

ValueError: Variable RNN/BasicLSTMCell/Linear/Matrix already exists, disallowed. Did you mean to set reuse=True in VarScope? 

pointing at line with outputs2, states2

resetting graph doesn't help in a slightest bit. If any additional information is needed to help resolve the issue, I will gladly provide it.

Answer 1

TensorFlow's RNN code uses "variable scopes" to manage the creation and sharing of variables, and in this case it cannot tell whether you want to create a new set of variables for the second RNN, or reuse an old set of variables.

Assuming that you want the weights for the two RNNs to be independent, you can address this error by wrapping each RNN in a differently-named with tf.variable_scope(name): block, as follows:

# ...
with tf.variable_scope("first_lstm"):
  lstm_cell = rnn_cell.BasicLSTMCell(ukryta_warstwa, forget_bias=1.0)
  _X = tf.split(0, liczba_krokow, _X)

  outputs, states = rnn.rnn(lstm_cell, _X, initial_state=_istate)

with tf.variable_scope("second_lstm"):
  lstm_cell_2 = rnn_cell.BasicLSTMCell(ukryta_warstwa, forget_bias=1.0)
  outputs2, states2 = rnn.rnn(lstm_cell_2, outputs, initial_state=_istate_2)
# ...