tensorflow/tfjs-node - Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type

Question

When trying to run @tensorflow-models/face-landmarks-detection with @tensorflow/tfjs-node on a nodejs server using express.js, I am getting the error: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type.

I am sending a base64 encoded string of an image to the server then trying to convert it to a tensor to pass into the model for prediction.

When I try to convert it to a tensor it returns the above-mentioned error. The image is of type jpeg. I have also tried changing the extension to JPEG just in case that was the issue.

I installed the packages with the following command (maybe order matters?) npm i --save @tensorflow-models/face-landmarks-detection @tensorflow/tfjs-node @tensorflow/tfjs-node-gpu

And am importing them as such:

const tf = require("@tensorflow/tfjs-node");
const faceLandmarksDetection = require("@tensorflow-models/face-landmarks-detection");

My base64 string: data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/4gIcS... (too long to past all here)

I initially tried converting the base64 string using node's native Buffer.from(str, 'base64)

     try {
      const model = await faceLandmarksDetection.load(
        faceLandmarksDetection.SupportedPackages.mediapipeFacemesh
      );
      const b = Buffer.from(dataObj.data_url, "base64");
      const tensor = tf.node.decodeImage(b);
      console.log("Tensor ", tensor);
      const preds = await model.estimateFaces({
        input: tensor,
      });
      console.log(preds);
    } catch (err) {
      console.log(err);
    }

I tried converting it to an Uint8Array using the package base64-arraybuffer According to the tensorflow node docs the image should be an encoded image in an Uint8Array.

     try {
      const model = await faceLandmarksDetection.load(
        faceLandmarksDetection.SupportedPackages.mediapipeFacemesh
      );
      const b = bBuffer.decode(dataObj.data_url);
      const tensor = tf.node.decodeImage(b); // code throwing error
      console.log("Tensor ", tensor);
      const preds = await model.estimateFaces({
        input: tensor,
      });
      console.log(preds);
    } catch (err) {
      console.log(err);
    }

I also tried passing in the raw base64 to the tf.node.decodeImage method. All seem to be returning the same error!

Any ideas why this would be happening?

Answer 1

Managed to resolve it with trimming the base64 string before passing it into the Buffer method:

      const img = dataObj.data_url.replace(
        /^data:image\/(png|jpeg);base64,/,
        ""
      );
      const b = Buffer.from(img, "base64");
      const tensor = tf.node.decodeImage(b, 3);