How to mix / combine multiple WebRTC media streams (screen capture + webcam) into a single stream?

Question

I have a live screen capture media stream returned from getDisplayMedia(),
and a live webcam media stream returned from getUserMedia().

I currently render the webcam video on top of the screen share video, to create a picture-in-picture effect:

I want to mix / combine them into one video stream, in order to render it inside a single HTML video element.
I want to keep both streams active & live just as if they were two separate videos rendering two different media streams.
I also need to maintain the streams positions - keeping the webcam stream small and on top of the screen share stream. It's also really important for me to keep the original resolution and bitrate.

How can I do that?

Answer 1

You can draw both video streams on an HTMLCanvasElement, and then create a MediaStream from this HTMLCanvasElement calling its captureStream method.

To draw the two video streams, you'd have to read them in <video> elements and drawImage() these <video> elements onto the canvas in a timed loop (e.g requestAnimationFrame can be used for this timed loop).

async function getOverlayedVideoStreams( stream1, stream2 ) {
  // prepare both players
  const vid1 = document.createElement("video");
  const vid2 = document.createElement("video");
  vid1.muted = vid2.muted = true;
  vid1.srcObject = stream1;
  vid2.srcObject = stream2;
  await Promise.all( [
    vid1.play(),
    vid2.play()
  ] );
  // craete the renderer
    const canvas = document.createElement("canvas");
  let w = canvas.width = vid1.videoWidth;
  let h = canvas.height = vid1.videoHeight;
  const ctx = canvas.getContext("2d");

  // MediaStreams can change size while streaming, so we need to handle it
  vid1.onresize = (evt) => {
    w = canvas.width = vid1.videoWidth;
    h = canvas.height = vid1.videoHeight;
  };
  // start the animation loop
  anim();
  
  return canvas.captureStream();
  
  function anim() {
    // draw bg video
        ctx.drawImage( vid1, 0, 0 );
    // caculate size and position of small corner-vid (you may change it as you like)
    const cam_w = vid2.videoWidth;
    const cam_h = vid2.videoHeight;
    const cam_ratio = cam_w / cam_h;
    const out_h = h / 3;
    const out_w = out_h * cam_ratio;
    ctx.drawImage( vid2, w - out_w, h - out_h, out_w, out_h );
    // do the same thing again at next screen paint
    requestAnimationFrame( anim );
  }
}

Live demo as a glitch since StackSnippets won't allow capture APIs.