Keras: input shape of a dense layer

Question

I am reading the tutorial at: https://www.datacamp.com/community/tutorials/generative-adversarial-networks

Where there is a network called generator with the following architecture:

generator = Sequential()

generator.add(Dense(256, input_dim=64, kernel_initializer=initializers.RandomNormal(stddev=0.02)))
generator.add(LeakyReLU(0.2))

generator.add(Dense(512))
generator.add(LeakyReLU(0.2))

generator.add(Dense(1024))
generator.add(LeakyReLU(0.2))

# last layer output a 28x28 image
generator.add(Dense(784, activation='tanh'))
generator.compile(loss='binary_crossentropy', optimizer=optimizer)

The input of the generator is called noise like below:

noise = np.random.normal(0, 1, size=[batch_size, input_dim])
print (noise.shape)

The shape of noise is:

noise shape:  (128, 64)

So we have:

generated_images = generator.predict(noise)

Here is what confuses me: the input_dim of the generator is already specified as 64 in the first Dense layer, why is it allowed to input noise with the shape (128,64)?

I understand the idea is to compute 128 elements in a batch each time. My question is what exactly the function predict can take? The Keras document (https://keras.io/models/sequential/) has the following:

predict(x, batch_size=None, verbose=0, steps=None)

x: The input data, as a Numpy array (or list of Numpy arrays if the model has multiple inputs).

But how do we know which dimension in x should be batch_size and which dimension in x should be input_dim? What happen if the noise shape = (64,128) ? or (64, 256, 256) or (256, 256, 64)? which ones are allowed? Is there a hidden document somewhere? Thanks!

Answer 1

Keras models expect the first dimension of your data to be the batch dimension. You can have a look at the docs on the Input layers from the functional API. This layer has a shape argument as well as an batch_shape argument. Both work, but the latters allow to explicitly define a batch shape.

To answer your questions:

• noise shape = (64,128) would give an error because the expected input dim is 64 and you provide 128 (the 64 in the first dimension does not matter here as the batch size can be anything and does not change the model architecture)
• (64, 256, 256) and (256, 256, 64) would give you an error because an input with two dimensions (+ 1 batch dimension) is provided, where the model expects 1-dimensional input (+ 1 batch dimension).

Also check out the docs of the Sequential API:

Pass an input_shape argument to the first layer. This is a shape tuple (a tuple of integers or None entries, where None indicates that any positive integer may be expected). In input_shape, the batch dimension is not included.

If you ever need to specify a fixed batch size for your inputs (this is useful for stateful recurrent networks), you can pass a batch_size argument to a layer. If you pass both batch_size=32 and input_shape=(6, 8) to a layer, it will then expect every batch of inputs to have the batch shape (32, 6, 8).