convolutional neural network for object localization

Question

I saw in deep learning course by Andrew Ng a way to localize single object on image : https://www.youtube.com/watch?v=GSwYGkTfOKk . As I understand it, you can for example bound a point to specific part of the object, take coordinates: x, y as labels y and train CNN.

I wanted to train a CNN neural network to localize my eyes (not clasiffication). I took 200 photos of me: 60x60 pixels in gray scale. I labeld left and right eye, Each coordinate of labeled eye was normalized to 0-1. The y label is : [x of eye1, y of eye1, x of eye2, y of eye2]. I used SGD optimazer with mse loss and in the output layer sigmoid function.

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(64, (3,3), input_shape= (60,60, 1)))
model.add(tf.keras.layers.Activation('relu'))
model.add(tf.keras.layers.MaxPool2D(pool_size=(2,2)))

model.add(tf.keras.layers.Conv2D(32, (3,3)))
model.add(tf.keras.layers.Activation('relu'))
model.add(tf.keras.layers.MaxPool2D(pool_size=(2,2)))

model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(64, activation='relu'))

model.add(tf.keras.layers.Dense(4, activation='sigmoid'))
sgd= tf.keras.optimizers.SGD(lr = 0.01)
model.compile(loss = 'mean_squared_error', optimizer=sgd, metrics=['accuracy'])
model.fit(x,y, batch_size=3, epochs=15, validation_split=0.2)

It didnt work for this task, so what is the way to solve this problem? I saw somewhere: apply CNN to image (I suppose without dense layers), then on flatten data from CNN use linear regression for each x/y coordinate (multivariable logistic regression). Is this a solution ? As I understand it, I would feed each image into Conv and MaxPool layers, then Flatten and then i feed the data to lin. regression and train it, but I have no idea how to do this in keras. I am new in this field, so any idea helps me.

Answer 1

First of all, a couple of observations with regard to your code.

Since the last layer contanins more than 2 neurons, the activation function that you have to use is softmax , not sigmoid (note that this is in the case of classification, not regression).

You should only use sigmoid when you are doing binary classification, but not when you have more than two classes (note that you can also use softmax for 2 classes, however it is not necessarily recommended from a small computational overhead viewpoint).

Your problem is both a regression and classification one!.

The first layer of your convolutional neural network contains 64 feature maps, with each size of the kernel 3x3. Although the way you feed the images to your neural network is correct, you only feed the grayscale image, not the x1,x2,y1,y2 coordinates.

For an ANN with regression, take a look at this tutorial: https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/.

Your intuition is correct; object detection neural networks replace fully connected layers with convolutional ones. Yann LeCun even states that fully connected layers should not be a part of CNNs.

Since you are new to this field, I would recommend to adopt the following pipeline.

1) Find an online github model written in your preferred deep learning library(Keras/PyTorch/TensorFlow etc).

2) Follow the instructions/tutorial in order to reproduce the results obtained by the github user.

3) By means of the latter you should also understand the code / get a good intuitive grasp.

4) Adapt the model to the problem that you need.

One place where you could start is here (this is object detection - detect multiple objects and also of different categories) : https://github.com/pierluigiferrari/ssd_keras.

If you have further questions, please write them down, I would be glad to be of assistance!