How to fix the 'ValueError: input tensor must have rank 4'?

Question

Currently I'm trying to run a CNN in combination with an LSTM model for a video classification, but after a search on Google and Stackoverflow I wasn't able to find the solution for my problem

Below is the whole code:

#Importing libraries 
from keras.preprocessing.image import ImageDataGenerator 
from keras.models import Sequential 
from keras.layers import Conv2D, MaxPooling2D, LSTM, TimeDistributed
from keras.layers import Activation, Dropout, Flatten, Dense 
from keras import backend as K

#Shape of the image, based on 1920x1080
img_width, img_height = 224, 135

#Location of the frames split in a train and test folder
train_data_dir = './train'
validation_data_dir = './test'

#Data information
nb_train_samples = 46822
nb_validation_samples = 8994
timesteps = 1
epochs = 10
batch_size = 30

input_shape = (img_width, img_height, 3)

model = Sequential()
# define CNN model
model.add(TimeDistributed(Conv2D(132, (3, 3), input_shape=input_shape, activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size = (2, 2))))
model.add(TimeDistributed(Flatten()))
# define LSTM model
model.add(LSTM(132, return_sequences=True))
model.add(LSTM(132, return_sequences=True))
model.add(LSTM(132, return_sequences=True))
model.add(LSTM(132, return_sequences=True))
model.add(Dense(3, activation='softmax'))

model.build(input_shape)

model.summary()

model.compile(loss ='categorical_crossentropy', optimizer ='rmsprop', metrics =['accuracy']) 

model.fit_generator(train_generator, steps_per_epoch = nb_train_samples // batch_size, epochs = epochs, validation_data = validation_generator, validation_steps = nb_validation_samples // batch_size)

train_datagen = ImageDataGenerator(rescale = 1. / 255, shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True) 

test_datagen = ImageDataGenerator(rescale = 1. / 255) 

train_generator = train_datagen.flow_from_directory(train_data_dir, target_size =(img_width, img_height), batch_size = batch_size, class_mode ='categorical') 

validation_generator = test_datagen.flow_from_directory(validation_data_dir, target_size =(img_width, img_height), batch_size = batch_size, class_mode ='categorical')  

The error that occures when running this is:

Traceback (most recent call last):
File "CNNLSTM.py", line 36, in
model.build(input_shape)
......
ValueError: input tensor must have rank 4

I've added the model.build(input_shape) to avoid this error:

ValueError: This model has not yet been built. Build the model first by calling build() or calling fit() with some data. Or specify input_shape or batch_input_shape in the first layer for automatic build.

But as is visible in the code, I've applied input_shape in the first line of the model.

Hopefully someone here can point out what I'm doing wrong.

Answer 1

There are three points you should consider:

1. You mentioned you are doing video classification. Therefore, the input of the model is a set of images/frames. So the input shape (i.e. one sample's shape) is:

   input_shape = (n_frames, img_width, img_height, 3)
   

2. The first layer of your model is TimeDistributed wrapper which wraps the Conv2D layer. Therefore, you must set the input_shape argument for this layer instead:

   model.add(TimeDistributed(Conv2D(132, (3, 3), activation='relu'), input_shape=input_shape))
   

3. The build method expects the batch shape as argument, not the shape of a single input sample. Therefore, you should write:

   model.build((None,) + input_shape)
   

   However, if you address the point #2 then you DON'T need to call build method at all.