How to access current video frame in Expo Camera?

Question

I'm trying to access the current video frame in React Native. I was able to do the same with 'react-webcam' while using React.js with the below code.

import React from "react";
import Webcam from "react-webcam";

function MyFunc () {

  const cameraRef = useRef(null);

  const processFrame = async() => {
    if (cameraRef.current.video){
      const img = cameraRef.current.video;
      // Code to process.
    };
    setTimeout(() => processFrame(), 500)
  };

  React.useEffect(() => {
    processFrame();
  }, [])

  return (
    <Webcam
      align="center"
      audio={false}
      mirrored={false}
      id="img"
      ref={cameraRef}
      style={{display: "none"}}
    />
  );
};

My current code in React Native using Expo Camera is -

import React, { useState, useEffect, useRef } from "react";

import { Camera } from "expo-camera";

export function MyFunc () {

  const cameraRef = useRef(null);

  const processFrame = async () => {
    const img = cameraRef.current.video;
    console.log(img); // This prints undefined

    setTimeout(() => processFrame(), 500)
  };

  useEffect(() => {
    processFrame();
  }, []);

  return (
    <Camera
      ref={cameraRef}
      type={Camera.Constants.Type.front}
      style={{opacity: 0, width:1, height:1}}
    />
  );
};

Please let me know how can I access the current video frame without using the asyncTakePicture if possible.

Answer 1

You need a real-time working camera which is Tensorflow Camera (build from expo-camera)

cameraWithTensors (CameraComponent), A higher-order-component (HOC) that augments the Expo.Camera component with the ability to yield tensors representing the camera stream.

Because the camera data will be consumed in the process, the original camera component will not render any content. This component provides options that can be used to render the camera preview.

Notably the component allows on-the-fly resizing of the camera image to smaller dimensions, this speeds up data transfer between the native and javascript threads immensely.

For more info: Tensorflow React Native API