Green Screen Effect with Videos in GMetri XR Experiences

 
 

The use of Green Screens has come a long way in the past decade. Technology that was primarily used in hollywood movies is now being used by everyone, including our favorite youtubers and insta-celebs. Here we'll go through the entire process required to enable green screen effect on your XR experience.

Before we dive into how to enrich virtual reality with green screen effect, let's familiarize ourselves with a few key terms:

Green Screen

Refers to a colored background that you want to remove from your shot. This is usually single colored backdrop that can be of any color. Usually, a bright green color is used because it is the furthest away from a human's skin color. In past times, blue colored screens were also popular. When there is a green colored prop required in the shot, a bright blue colored screen is used instead.

Chroma Key

Goes hand-in-hand with green screen. It is the technique of compositing two images/videos together based on color hues. All colors have a chroma component, hence the name.

Keying

Refers to the process of removing the green screen background from the shot while post processing the shot. The portions of the video which match the pre-selected color are replaced by transparent pixels which can be later filled in with another image or video.

State of XR and Transparent Video Content on the web

Ever since chrome released support for transparent video in webm format, there has been a surge of websites using transparent webm videos to give that oomph factor to their content. The virtual reality community hasn't stayed far behind.

Three.js and babylon.js are two of the most popular 3D rendering engines on the web. Both of these support native HTML based video rendering using WebGL. Both engines achieve this in XR by drawing the frames of a video onto a canvas. And thus began the journey of using videos in VR experiences, creating interactive and gamified content to the masses. We won't go into the details on how they did so in this post.

Once a great man said:

With great power, comes great responsibility.

Even though webm video format is supported in major browsers, it still suffers from potential failure points.

  • The internal spec is still shaky. Transparency doesn't work at times on certain devices.

  • Encoding a webm video is a hassle. Getting the right codec and media settings can be tricky. Encoding a webm video doesn't follow the well known approaches used by designers and VFX artists. It requires special knowledge of ffmpeg and/or blender.

  • Safari doesn't support transparency in webm videos!