LSTM Neural Network Input/Output dimensions error

Question

I am fairly new to TensorFlow and LSTM architecture. I have an issue with figuring out input and output (x_train, x_test, y_train, y_test) for my dataset.

Original shapes of my inputs:

• x_train: (366,4)
• x_test: (104,4)
• y_train: (366,)
• y_test: (104,)

The y_train and y_test are a series of stock prices. The x_train and x_test are four features that I want to use to predict the stock prices.

# Splitting the training and testing data

train_start_date = '2010-01-08'
train_end_date = '2017-01-06'
test_start_date = '2017-01-13'
test_end_date = '2019-01-04'

train = df.ix[train_start_date : train_end_date]
test = df.ix[test_start_date:test_end_date]


X_test = sentimentScorer(test)
X_train = sentimentScorer(train)

Y_test = test['prices'] 
Y_train = train['prices']

#Conversion in 3D array for LSTM INPUT

X_test = X_test.reshape(1, 104, 4)
X_train = X_train.reshape(1, 366, 4)





model = Sequential()

model.add(LSTM(128, input_shape=(366,4), activation='relu', 
return_sequences=True))
model.add(Dropout(0.2))

model.add(LSTM(128, activation='relu'))
model.add(Dropout(0.1))

model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(10, activation='softmax'))

opt = tf.keras.optimizers.Adam(lr=0.001, decay=1e-6)

# Compile model
model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'],
)

model.fit(X_train,
          Y_train,
          epochs=3,
          validation_data=(X_test, Y_test))

This is the error generated:

> --------------------------------------------------------------------------- ValueError                                Traceback (most recent call
> last) <ipython-input-101-fd4099583529> in <module>
>      65           Y_train,
>      66           epochs=3,
> ---> 67           validation_data=(X_test, Y_test))
> 
> c:\users\talal\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\keras\engine\training.py
> in fit(self, x, y, batch_size, epochs, verbose, callbacks,
> validation_split, validation_data, shuffle, class_weight,
> sample_weight, initial_epoch, steps_per_epoch, validation_steps,
> **kwargs)    1507         steps_name='steps_per_epoch',    1508         steps=steps_per_epoch,
> -> 1509         validation_split=validation_split)    1510     1511     # Prepare validation data.
> 
> c:\users\talal\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\keras\engine\training.py
> in _standardize_user_data(self, x, y, sample_weight, class_weight,
> batch_size, check_steps, steps_name, steps, validation_split)
>     991       x, y = next_element
>     992     x, y, sample_weights = self._standardize_weights(x, y, sample_weight,
> --> 993                                                      class_weight, batch_size)
>     994     return x, y, sample_weights
>     995 
> 
> c:\users\talal\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\keras\engine\training.py
> in _standardize_weights(self, x, y, sample_weight, class_weight,
> batch_size)    1110         feed_input_shapes,    1111        
> check_batch_axis=False,  # Don't enforce the batch size.
> -> 1112         exception_prefix='input')    1113     1114     if y is not None:
> 
> c:\users\talal\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\keras\engine\training_utils.py
> in standardize_input_data(data, names, shapes, check_batch_axis,
> exception_prefix)
>     314                            ': expected ' + names[i] + ' to have ' +
>     315                            str(len(shape)) + ' dimensions, but got array '
> --> 316                            'with shape ' + str(data_shape))
>     317         if not check_batch_axis:
>     318           data_shape = data_shape[1:]
> 
> ValueError: Error when checking input: expected lstm_18_input to have
> 3 dimensions, but got array with shape (366, 4)

Answer 1

Your code is almost fine.

Your y_test and y_train should be an array with one element or array of shape (1,1), it doesn't matter.

Your input shape is wrong though, first LSTM should be:

model.add(LSTM(128, input_shape=(None,4), activation='relu', return_sequences=True))

Notice None, as your test and train sequences length are different, you cannot specify it (and Keras accepts first dimension unspecified). Error was due to lengths of 366 and 104 respectively. If you want to use batches with RNNs you should perform zero-padding with keras.preprocessing.sequence.pad_sequences.

No need to specify input_shape with batch, rest of the network should be fine.

And if you are performing regression, not classification, as is the case probably, you should perform two last steps written by @Ankish Bansal, e.g. changing loss to mean squared error and making last layer output 1 value instead of 10.