Creating a video with AVVideoWriter using CMSampleBuffers extracted from another video

Question

I am trying to create and save a video using sample buffers taken from another video. I'm using Swift Playground and have the following setting enabled so that Dispatch works properly.

PlaygroundPage.current.needsIndefiniteExecution = true

An asset reader is then declared and used to read the sample buffers. The sample buffers are valid (converting a random buffer to a UIImage and displaying it shows the correct frame).

// A sample buffer array is populated from a source video.
// Source video is just a three seconds 720p video at 30fps (m4v).

var sampleBuffer: CMSampleBuffer? = assetReaderOutput.copyNextSampleBuffer()
var sourceSampleBufferArray = [CMSampleBuffer]()
while (sampleBuffer != nil) {
    sampleBuffer = assetReaderOutput.copyNextSampleBuffer()
    if (sampleBuffer != nil) {
        sourceSampleBufferArray.append(sampleBuffer!)
    }
}

Now I am attempting to save the sample buffers as a new video using AVAssetWriter.

let documentsPath = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString
let videoOutputURL = URL(fileURLWithPath: documentsPath.appendingPathComponent("AssembledVideo.m4v"))

do {
    try FileManager.default.removeItem(at: videoOutputURL as URL)
}
catch {}

let assetWriter = try AVAssetWriter(outputURL: videoOutputURL as URL, fileType: AVFileTypeAppleM4V)
let outputSettings: Dictionary<String, AnyObject> = [
    AVVideoCodecKey : AVVideoCodecH264 as AnyObject,
    AVVideoWidthKey : 1280 as AnyObject,
    AVVideoHeightKey : 720 as AnyObject
];
let assetWriterInput = AVAssetWriterInput(mediaType: AVMediaTypeVideo, outputSettings: outputSettings)
assetWriter.add(assetWriterInput)
let mediaQueue = DispatchQueue(label: "DispatchQueue")

if assetWriter.startWriting() {
    var counter = sourceSampleBufferArray.count
    assetWriter.startSession(atSourceTime: kCMTimeZero)
    assetWriterInput.requestMediaDataWhenReady(on: mediaQueue) {
        while assetWriterInput.isReadyForMoreMediaData {
            if counter > 0 {
                assetWriterInput.append(sourceSampleBufferArray[counter - 1])
                counter = counter - 1
            }
            else {
                assetWriterInput.markAsFinished()
                assetWriter.finishWriting(completionHandler: {
                    print(assetWriter.outputURL)
                })
                break
            }
        }
    }
}

I get no errors and the code seems to run without problems except that the video created is zero bytes. Any suggestions or help to figure out why the video is not properly saved would be appreciated. Thanks :)

Answer 1

Each frame/sample buffer has its own presentation timestamp. These are probably incorrect when transplanted into your new video. You can examine the timestamps with

let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)

If you're only inserting one video into your output video, the solution could be as simple as changing your output video's timeline origin, with:

let startTime = CMSampleBufferGetPresentationTimeStamp(firstInputSampleBuffer)
assetWriter.startSession(atSourceTime: startTime)

If you're inserting multiple videos, with multiple incompatible timelines, you will need to make a (shallow!) copy of the individual sample buffers with correct presentation timestamps that you calculate, using

CMSampleBufferCreateCopyWithNewTiming()