How to correctly pass YUV_420_888 Image Buffer from Java through JNI to OpenCV, accounting for stride/padding

Question

I'm trying to convert the image data from an Android device from YUV_420_888 to an RGB matrix on the C++ side. On some devices, this is working flawlessly. On a Note 10, the image comes out looking like this:

My guess here is that the stride is causing this issue. How do I remove this extra data and then pass the correct buffer through JNI?

Here is the Java code:

IntBuffer rgb = image.getPlanes()[0].getBuffer().asIntBuffer();
NativeLib.passImageBuffer(rgb);

And here is the C++ code:

cv::Mat outputRGB;
cv::cvtColor(cv::Mat(height+height/2, width, CV_8UC1, inputRGB), outputRGB, CV_YUV2BGR_NV21);

I've tried some different image formats on the C++ side, but they all come back with the same band on the side of the screen.

I've implemented this answer, in order to remove the extra padding, but the image that is passed ends up being completely green. Do some corresponding edits need to be made to the C++ code? I've tried using a 3 channel format, but that crashes at runtime. I'm thinking that since passing the buffer works with the 1 channel matrix on phones that have 8 bits per pixel, that it should be possible to do that with the note 10?

            Image.Plane Y = image.getPlanes()[0];
            Image.Plane U = image.getPlanes()[1];
            Image.Plane V = image.getPlanes()[2];

            int[] rgbBytes = new int[image.getHeight()*image.getWidth()*4];
            int idx = 0;
            ByteBuffer yBuffer = Y.getBuffer();
            int yPixelStride = Y.getPixelStride();
            int yRowStride = Y.getRowStride();
            ByteBuffer uBuffer = U.getBuffer();
            int uPixelStride = U.getPixelStride();
            int uRowStride = U.getRowStride();
            ByteBuffer vBuffer = V.getBuffer();
            int vPixelStride = V.getPixelStride();
            int vRowStride = V.getRowStride();

            ByteBuffer rgbBuffer = ByteBuffer.allocateDirect(rgb.limit());
            for (int row = 0; row < image.getHeight(); row++) {
              for (int col = 0; col < image.getWidth(); col++) {
                int y = yBuffer.get(col*yPixelStride + row*yRowStride) & 0xff;
                int u = uBuffer.get(col/2*uPixelStride + row/2*uRowStride) & 0xff;
                int v = vBuffer.get(col/2*vPixelStride + row/2*vRowStride) & 0xff;

                int y1 = ((19077 << 8) * y) >> 16;
                int r = (y1 + (((26149 << 8) * v) >> 16) - 14234) >> 6;
                int g = (y1 - (((6419 << 8) * u) >> 16) - (((13320 << 8) * v) >> 16) +  8708) >> 6;
                int b = (y1 + (((33050 << 8) * u) >> 16) - 17685) >> 6;

                if (r < 0) r = 0;
                if (g < 0) g = 0;
                if (b < 0) b = 0;
                if (r > 255) r = 255;
                if (g > 255) g = 255;
                if (b > 255) b = 255;
                byte pixel = (byte)(0xff000000 + b + 256 * (g + 256 * r));
                rgbBuffer.put(pixel);
              }
            }

Answer 1

Look at this repo https://github.com/quickbirdstudios/yuvToMat/

It supports different formats (YUV420, NV12) and variety of pixel and row strides.