AVSampleBufferDisplayLayer not rendering on device

Question

I'm struggling with AVSampleBufferDisplayLayer on iOS. I want to display a CVPixelBuffer using this layer, but i'm not able to get it work on actual iOS device. In my sample app i tried following code to display one color pixel buffer:

@implementation ViewController {
    AVSampleBufferDisplayLayer *videoLayer;
}

- (void)viewDidLoad {
    [super viewDidLoad];

    videoLayer = [[AVSampleBufferDisplayLayer alloc] init];
    videoLayer.frame = CGRectMake(50, 50, 300, 300);
    videoLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;

    [self.view.layer addSublayer:videoLayer];
}

@implementation ViewController {
    AVSampleBufferDisplayLayer *videoLayer;
}

- (void)viewDidLoad {
    [super viewDidLoad];

    videoLayer = [[AVSampleBufferDisplayLayer alloc] init];
    videoLayer.frame = CGRectMake(50, 50, 300, 300);
    videoLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;

    [self.view.layer addSublayer:videoLayer];
}

- (void)viewDidAppear:(BOOL)animated {
    [super viewDidAppear:animated];
    [self startVideo];
}

- (void)startVideo {
    [self drawPixelBuffer];
    [NSTimer scheduledTimerWithTimeInterval:0.1 target:self selector:@selector(drawPixelBuffer) userInfo:nil repeats:YES];
}

- (void)drawPixelBuffer {

    int imageSize = 100;

    static const uint8_t pixel[] = {0x00, 0xAA, 0xFF, 0xFF};

    NSMutableData *frame = [NSMutableData data];

    for (int i = 0; i < imageSize * imageSize; i++) {
        [frame appendBytes:pixel length:4];
    }

    CVPixelBufferRef pixelBuffer = NULL;

    CVPixelBufferCreateWithBytes(NULL, imageSize, imageSize, kCVPixelFormatType_32BGRA, [frame bytes], imageSize * 4, NULL, NULL, NULL, &pixelBuffer);

    CMSampleBufferRef sampleBuffer = [self sampleBufferFromPixelBuffer:pixelBuffer];

    if (sampleBuffer) {

        [videoLayer enqueueSampleBuffer:sampleBuffer];
        CFRelease(sampleBuffer);

    }

}

- (CMSampleBufferRef)sampleBufferFromPixelBuffer:(CVPixelBufferRef)pixelBuffer {

    CMSampleBufferRef sampleBuffer = NULL;
    OSStatus err = noErr;
    CMVideoFormatDescriptionRef formatDesc = NULL;
    err = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, &formatDesc);

    if (err != noErr) {
        return nil;
    }

    CMSampleTimingInfo sampleTimingInfo = kCMTimingInfoInvalid;

    err = CMSampleBufferCreateReadyWithImageBuffer(kCFAllocatorDefault, pixelBuffer, formatDesc, &sampleTimingInfo, &sampleBuffer);

    if (sampleBuffer) {
        CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, YES);
        CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);
        CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);
    }

    if (err != noErr) {
        return nil;
    }

    formatDesc = NULL;

    return sampleBuffer;

}

@end

This is working without any problem in iOS simulator but it's not working on real device (nothing is rendered). The video layer's error property is always nil and status is always equal AVQueuedSampleBufferRenderingStatusRendering.

Thanks for any help.

Answer 1

The graphics implementation in the simulator is far more robust, and often you can get away with things that will not work on device. There are two common causes:

Pixel buffers should be backed by an IOSurface

You're mapping that buffer directly via CVPixelBufferCreateWithBytes. Try again using CVPixelBufferCreate with the kCVPixelBufferIOSurfacePropertiesKey attribute set to an empty dictionary.

CVPixelBufferCreate(
    NULL,
    imageSize,
    imageSize,
    kCVPixelFormatType_32BGRA,
    (__bridge CFDictionaryRef)@{
        (id)kCVPixelBufferIOSurfacePropertiesKey: @{}
    },
    &pixelBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
void *bytes = CVPixelBufferGetBaseAddress(pixelBuffer);
// Write image data directly to that address
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);

In reality, whatever is generating those pixels should be writing directly into a CVPixelBufferRef whenever possible.

Devices will not support some formats

This seems extremely unlikely for kCVPixelFormatType_32BGRA, but I have seen simulator-only support with others such as kCVPixelFormatType_422YpCbCr8. In these cases it must be converted to a compatible format first, or a custom renderer must be implemented (OpenGL, Metal, etc).