Converting cv::Mat to MTLTexture

Question

An intermediate step of my current project requires conversion of opencv's cv::Mat to MTLTexture, the texture container of Metal. I need to store the Floats in the Mat as Floats in the texture; my project cannot quite afford the loss of precision.

This is my attempt at such a conversion.

- (id<MTLTexture>)texForMat:(cv::Mat)image context:(MBEContext *)context
{

  id<MTLTexture> texture;
  int width = image.cols;
  int height = image.rows;
  Float32 *rawData = (Float32 *)calloc(height * width * 4,sizeof(float));
  int bytesPerPixel = 4;
  int bytesPerRow = bytesPerPixel * width;
  float r, g, b,a;


  for(int i = 0; i < height; i++)
  {
    Float32* imageData = (Float32*)(image.data + image.step * i);
    for(int j = 0; j < width; j++)
    {
      r = (Float32)(imageData[4 * j]);
      g = (Float32)(imageData[4 * j + 1]);
      b = (Float32)(imageData[4 * j + 2]);
      a = (Float32)(imageData[4 * j + 3]);

      rawData[image.step * (i) + (4 * j)] = r;
      rawData[image.step * (i) + (4 * j + 1)] = g;
      rawData[image.step * (i) + (4 * j + 2)] = b;
      rawData[image.step * (i) + (4 * j + 3)] = a;
    }
  }


  MTLTextureDescriptor *textureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA16Float
                                                                                               width:width
                                                                                              height:height
                                                                                           mipmapped:NO];
  texture = [context.device newTextureWithDescriptor:textureDescriptor];
  MTLRegion region = MTLRegionMake2D(0, 0, width, height);
  [texture replaceRegion:region mipmapLevel:0 withBytes:rawData bytesPerRow:bytesPerRow];

  free(rawData);
  return texture;
}

But it doesn't seem to be working. It reads zeroes every time from the Mat, and throws up EXC_BAD_ACCESS. I need the MTLTexture in MTLPixelFormatRGBA16Float to keep the precision.

Thanks for considering this issue.

Answer 1

One problem here is you’re loading up rawData with Float32s but your texture is RGBA16Float, so the data will be corrupted (16Float is half the size of Float32). This shouldn’t cause your crash, but it’s an issue you’ll have to deal with.

Also as “chappjc” noted you’re using ‘image.step’ when writing your data out, but that buffer should be contiguous and not ever have a step that’s not just (width * bytesPerPixel).