tflite model with a Tensor.image as input instead of Tensor.buffer

Question

I'm new to machine learning and have (maybe a stupid) question. When I create a model for image classification in Keras

train = ImageDataGenerator().flow_from_directory(train_path, target_size = (128,128))test = ImageDataGenerator().flow_from_directory(test_path, target_size = (128,128))
model = Sequential()
num_classes = 131
model.add(Convolution2D(32, (3,3), activation = 'relu', padding='same', use_bias=False, input_shape=(128,128,3)))
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(32, (3,3), activation = 'relu', padding='same', use_bias=False))
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(64, (3,3), activation = 'relu', padding='same', use_bias=False))
model.add(MaxPool2D(pool_size=(2, 2)))


model.add(Flatten())
model.add(Dense(128,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes,activation = 'softmax'))
model.summary()

and convert it to tflite

import tensorflow as tf
saved_model_dir = '~/'
tf.saved_model.save(model, saved_model_dir)

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()

with open('Fruits360ModelByKaggle.tflite', 'wb') as f:
  f.write(tflite_model)

labels = '\n'.join(sorted(train.class_indices.keys()))

with open('labels.txt', 'w') as f:
  f.write(labels)

Android Studio tells me, that the input the model expects is a TensorBuffer. I used the sample Project from these guys (Google Codelabs, which recognizes flowers and their model expects as input a TensorImage. So I tried to use my own model in this project and convert the bitmap to a TensorBuffer but failed big times.

 val tfImage = toBitmap(imageProxy)
            val byteBuffer: ByteBuffer = ByteBuffer.allocate(224*224*3)
            byteBuffer.rewind()
            if (tfImage != null) {
                tfImage.copyPixelsToBuffer(byteBuffer)
            }
            val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
            inputFeature0.loadBuffer(byteBuffer)

           results
            val outputs = myModel.process(inputFeature0)

Is there a way to create a model in Keras which accepts TensorImage as an input and can somebody help me with this. It would be much appreciated. Thanks so much!

Answer 1

Is there a way to create a model in Keras which accepts TensorImage as an input and can somebody help me with this.

No, you can't. But you do not need either. TFLite is model conversion optimized to work with HW other than desktops/servers and it has a bit specific way to consume data with lower latency via bytebuffers. TensorImage and TensorBuffer are wrappers from lite support library to provide you with simple pre-post processing instruments for simpler data feeding.
Piece of code provided by you converts data from camera to TensorImage of needed size. To take into account your model input size (128,128,3) your piece of code should be edited in next way:

 val tfImage = toBitmap(imageProxy)
            val byteBuffer: ByteBuffer = ByteBuffer.allocate(128*128*3)
            byteBuffer.rewind()
            if (tfImage != null) {
                tfImage.copyPixelsToBuffer(byteBuffer)
            }
            val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 128, 128, 3), DataType.FLOAT32)
            inputFeature0.loadBuffer(byteBuffer)

           results
            val outputs = myModel.process(inputFeature0)

Also check how outputs are processed.
UPD: Classification models takes as input image and returns guess class (and scores). This data requires other kind of container and TensorBuffer being used. Take a look inside of myModel and check what process method returns. Its possible it returns TensorBuffer and you need to read/postprocess it or maybe just class/score/something else.