Video composition with Media Foundation

Question

I'm trying to composite 2 videos into one (audio too) and save as a file. Say, show a source video on the left half of the final video and the ohter source on the right. Any encoding is fine. Not with DirectShow.

I suspected IMFVideoMixerControl may be related to this but unfortunately I'm too new to MF to detemine the right step to take in order to achieve this. If you could give me some guidance or if any example available on the web, please let me know. Thank you for your help.

Answer 1

IMFVideoMixerControl is used for rendering video. You're on the right track looking at the SourceReader.

Here's what I would do:

1. Create one IMFSourceReader for each video source.
2. Create an IMFSinkWriter and configure it with a frame width equal to the sum of the source reader frame widths.
   1. Use IMFSourceReader::GetCurrentMediaType to get the source media types.
   2. Use MFGetAttributeSize with GUID MF_MT_FRAME_SIZE to get the frame dimensions for each source type.
   3. Create a media type for the SinkWriter with MFCreateMediaType and use IMFMediaType::CopyAllItems to copy attributes from the source to sink media types.
   4. Use MFSetAttributeSize with GUID MF_MT_FRAME_SIZE to set the sink type's increased frame dimensions.
   5. Use IMFSinkWriter::AddStream to create a video stream identical to the source type, except for the width attribute
3. Call IMFSourceReader::ReadSample for each source, giving you one IMFSample for each source.
4. Allocate a new IMFSample, attaching a new IMFMediaBuffer with the increased frame width.
5. Use MFCopyImage to copy each of the source buffers to the corresponding side of the allocated media buffer.
6. Use IMFSinkWriter::WriteSample to write your IMFSample to the file sink.

See this sample for some basic SourceReader/SinkWriter processing, although this example uses a video capture source instead of a file. You can create a file SourceReader using MFCreateSourceReaderFromURL instead of MFCreateSourceReaderFromMediaSource.

Edit: I realized that you asked about audio, too. My answer only addresses compositing the video streams.