Generate Laplacian image by Apple-Metal MPSImageLaplacian

Question

I am trying to generate Laplacian image out of rgb CGImage by using metal laplacian.

The current code used:

if let croppedImage = self.cropImage2(image: UIImage(ciImage: image), rect: rect)?.cgImage {

  let commandBuffer = self.commandQueue.makeCommandBuffer()!

  let laplacian = MPSImageLaplacian(device: self.device)

  let textureLoader = MTKTextureLoader(device: self.device)

  let options: [MTKTextureLoader.Option : Any]? = nil

  let srcTex = try! textureLoader.newTexture(cgImage: croppedImage, options: options)

  let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: srcTex.pixelFormat, width: srcTex.width, height: srcTex.height, mipmapped: false)

  let lapTex = self.device.makeTexture(descriptor: desc)

  laplacian.encode(commandBuffer: commandBuffer, sourceTexture: srcTex, destinationTexture: lapTex!)

  let output = CIImage(mtlTexture: lapTex!, options: [:])?.cgImage

  print("output: \(output?.width)")


  print("") 
}

I suspect the problem is in makeTexture:

  let lapTex = self.device.makeTexture(descriptor: desc)

• the width and height of the lapTex in debugger are invalid although the desc and srcTex contains valid data including width and height.

Looks like order or initialisation is wrong but couldn't find what.

Does anyone has an idea what is wrong?

Thanks

Answer 1

There are a few things wrong here.

First, as mentioned in my comment, the command buffer isn't being committed, so the kernel work is never being performed.

Second, you need to wait for the work to complete before attempting to read back the results. (On macOS you'd additionally need to use a blit command encoder to ensure that the contents of the texture are copied back to CPU-accessible memory.)

Third, it's important to create the destination texture with the appropriate usage flags. The default of .shaderRead is insufficient in this case, since the MPS kernel writes to the texture. Therefore, you should explicitly set the usage property on the texture descriptor (to either [.shaderRead, .shaderWrite] or .shaderWrite, depending on how you go on to use the texture).

Fourth, it may be the case that the pixel format of your source texture isn't a writable format, so unless you're absolutely certain it is, consider setting the destination pixel format to a known-writable format (like .rgba8unorm) instead of assuming the destination should match the source. This also helps later when creating CGImages.

Finally, there is no guarantee that the cgImage property of a CIImage is non-nil when it wasn't created from a CGImage. Calling the property doesn't (necessarily) create a new backing CGImage. So, you need to explicitly create a CGImage somehow.

One way of doing this would be to create a Metal device-backed CIContext and use its createCGImage(_:from:) method. Although this might work, it seems redundant if the intent is simply to create a CGImage from a MTLTexture (for display purposes, let's say).

Instead, consider using the getBytes(_:bytesPerRow:from:mipmapLevel:) method to get the bytes from the texture and load them into a CG bitmap context. It's then trivial to create a CGImage from the context.

Here's a function that computes the Laplacian of an image and returns the resulting image:

func laplacian(_ image: CGImage) -> CGImage? {
    let commandBuffer = self.commandQueue.makeCommandBuffer()!

    let laplacian = MPSImageLaplacian(device: self.device)

    let textureLoader = MTKTextureLoader(device: self.device)
    let options: [MTKTextureLoader.Option : Any]? = nil
    let srcTex = try! textureLoader.newTexture(cgImage: image, options: options)

    let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: srcTex.pixelFormat,
                                                        width: srcTex.width,
                                                        height: srcTex.height,
                                                        mipmapped: false)
    desc.pixelFormat = .rgba8Unorm
    desc.usage = [.shaderRead, .shaderWrite]

    let lapTex = self.device.makeTexture(descriptor: desc)!

    laplacian.encode(commandBuffer: commandBuffer, sourceTexture: srcTex, destinationTexture: lapTex)

    #if os(macOS)
    let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()!
    blitCommandEncoder.synchronize(resource: lapTex)
    blitCommandEncoder.endEncoding()
    #endif

    commandBuffer.commit()
    commandBuffer.waitUntilCompleted()

    // Note: You may want to use a different color space depending
    // on what you're doing with the image
    let colorSpace = CGColorSpaceCreateDeviceRGB()
    // Note: We skip the last component (A) since the Laplacian of the alpha
    // channel of an opaque image is 0 everywhere, and that interacts oddly
    // when we treat the result as an RGBA image.
    let bitmapInfo = CGImageAlphaInfo.noneSkipLast.rawValue
    let bytesPerRow = lapTex.width * 4
    let bitmapContext = CGContext(data: nil,
                                  width: lapTex.width,
                                  height: lapTex.height,
                                  bitsPerComponent: 8,
                                  bytesPerRow: bytesPerRow,
                                  space: colorSpace,
                                  bitmapInfo: bitmapInfo)!
    lapTex.getBytes(bitmapContext.data!,
                    bytesPerRow: bytesPerRow,
                    from: MTLRegionMake2D(0, 0, lapTex.width, lapTex.height),
                    mipmapLevel: 0)
    return bitmapContext.makeImage()
}