using RNN with CNN in Keras

Question

beginner question

Using Keras I have a sequential CNN model that predicts an output of the size of [3*1] (regression) based on image(input).

How to implement RNN in order to add the output of the model as a second input to the next step. (so that we have 2 inputs: the image and the output of the previous sequence)?

model = models.Sequential()
model.add(layers.Conv2D(64, (3, 3), activation='relu', input_shape=X.shape[1:]))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dense(3, activation='linear'))

Answer 1

The easiest method I found was to directly extend Model. The following code will work in TF 2.0, but may not work in older versions:

class RecurrentModel(Model):
    def __init__(self, num_timesteps, *args, **kwargs):
        self.num_timesteps = num_timesteps
        super().__init__(*args, **kwargs)

    def build(self, input_shape):
        inputs = layers.Input((None, None, input_shape[-1]))
        x = layers.Conv2D(64, (3, 3), activation='relu'))(inputs)
        x = layers.MaxPooling2D((2, 2))(x)
        x = layers.Flatten()(x)
        x = layers.Dense(3, activation='linear')(x)
        self.model = Model(inputs=[inputs], outputs=[x])

    def call(self, inputs, **kwargs):
        x = inputs
        for i in range(self.num_timestaps):
            x = self.model(x)
        return x