Creating ad displaying a UIImage from raw BGRA data

Question

I'm collecting image data from the camera using this code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    // Called when a frame arrives
    // Should be in BGRA format
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);
    unsigned char *raw = (unsigned char *)CVPixelBufferGetBaseAddress(imageBuffer);

    // Copy memory into allocated buffer
    unsigned char *buffer = malloc(sizeof(unsigned char) * bytesPerRow * height);
    memcpy(buffer, raw, bytesPerRow * height);
    [self processVideoData:buffer width:width height:height bytesPerRow:bytesPerRow];

    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}

The processVideoData: method looks like this:

- (void)processVideoData:(unsigned char *)data width:(size_t)width height:(size_t)height bytesPerRow:(size_t)bytesPerRow
{
    dispatch_sync(dispatch_get_main_queue(), ^{

        CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, data, bytesPerRow * height, NULL);
        CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
        CGImageRef image = CGImageCreate(width, height, 8, 32, bytesPerRow, colorSpace, kCGBitmapByteOrderDefault | kCGImageAlphaLast, dataProvider, NULL, NO, kCGRenderingIntentDefault);

        // Set layer contents???
        UIImage *objcImage = [UIImage imageWithCGImage:image];
        self.imageView.image = objcImage;

        free(data);
        CGDataProviderRelease(dataProvider);
        CGColorSpaceRelease(colorSpace);
        CGImageRelease(image);
    });
}

No complaints, no leaks but nothing shows up in the image view it just stays blank (yes I have checked the outlet connection). Previously I had the bitmapInfo set just to kCGBitmapByteOrderDefault which was causing a crash when setting the image property of the image view however the image view would go dark which was promising just before the crash.

I summarised that the crash was due to the image being in BGRA not BGR so I set the bitmapInfo to kCGBitmapByteOrderDefault | kCGImageAlphaLast and that solved the crash but no image.

I realise that the image will look weird as the CGImageRef is expecting an RGB image and I'm passing it BGR but that should only result in a weird looking image due to channel swapping. I have also logged out the data that I'm getting and it seems to be in order something like: b:65 g:51 r:42 a:255 and the alpha channel is always 255 as expected.

I'm sorry if it's obvious but I can't work out what is going wrong.

Answer 1

You can use this flag combination to achieve the BGRA format:

kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst

You should prefer to use this solution, it will be a more performant way compared to an OpenCV conversion.

Here is a more common way to convert sourcePixelFormat to bitmapInfo:

sourcePixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
bitmapInfo = @{
    @(kCVPixelFormatType_32ARGB) : @(kCGBitmapByteOrder32Big | kCGImageAlphaNoneSkipFirst),
    @(kCVPixelFormatType_32BGRA) : @(kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst),
}[@(sourcePixelFormat)].unsignedIntegerValue;