Unity Video Webcam Backbuffer Tutorial

How to create a Unity 4.6+ application with a video/webcam backbuffer or background.

In this tutorial (which is demonstrated in the git repository) I will show you how to place a web camera video in the background of a Unity 4.6 application. This is required for video-see through Augmented Reality.

---

Acknowledgement: I borrowed a lot of text in the tutorial from this excellent tutorial from Jay May. However, this earlier tutorial is a little outdated and didn't work out of the box on Unity 4.6.

---

So I made some modifications...Here is my tutorial:

Theory: The approach we need for a video backbiter involves two Unity Cameras. The Main camera will be used for the virtual stuff...and will draw all the virtual objects in the scene. The second Unity camera, the Background Camera, will handle displaying the video backbuffer we need for an augmented reality magic lens view. The Background Camera will be set to only render a GUI layer. We will place a texture on this GUI layer, and at runtime, replace that texture with the WebCameraTexture.

OK, here we go:

1. Create a scene with a plane and a cube and sphere. This will be the virtual content that will be overlaid in front of the video or Augmented Reality View.

2. Create a new layer to store the background video texture.

Click on the "Layers" drop-down at the upper-right corner of the Unity editor window; from the layer drop down menu select the "Edit Layers..." option (last in the list). This should show the Tag Manager in the Inspector pane. The first 7 layers are Built in (reserved by the engine), so click next to where it says "User Layer 8" and type the name "Background". The layer number and the name you give it are NOT important for the purposes of this process, so feel free to use a different user layer or name if you want to.

3. Create a camera that will display the background video image. This is the camera that will handle the back buffer.

From the "GameObject" menu select "Camera". This will put a new camera in the scene, the position and orientation of this background camera is not important. Name the camera "Background Camera" in the Hierarchy pane.

4. Setup the Background Camera for rendering only the video image.

Click on the Background Camera in the Hierarchy pane to select it, and in the Inspector pane click on the "Camera" header to open the properties for the Camera component (if it's not already open). Set the following properties to these values (in this order):

Clear Flags = "Solid Color"

Culling Mask = "Nothing" (this unchecks all of the layers)

Culling Mask = "Background" (so only the background layer draws)

Depth = -1 (a lower depth than Main Camera so it will draw first)

Uncheck the "Flare Layer" and "Audio Listener" components. Keep the "GUILayer" component checked.

Near the top of the Inspector pane there is a choice box for "Layer", click this and select the "Background" layer you created in step 2.

4. Create the Background Image (backbiter) as a GUITexture.

NOTE: The GUITexture is a deprecated entity in Unity 4.6. I tried using a Canvas Image, but couldn't get it to work. Although the GUITexture is not a GameObject, it can still be added as a component (luckily)

From the "GameObject" menu at the top of the Unity IDE select "Create Empty" Rename the object "BackgroundImage" in the hierarchy pane. Click on it in the Hierarchy pane. In the "Component" Menu in Unity, Select "Add.." then search for GUITexture. Add the GUITexture as a component to the empty "BackgroundImage" object.

For the GUITexture component, all of the "Pixel Inset" and "Border" values should all be set to zero.

Under the "Transform" component properties set the Position to (0.5, 0.5, 0) and the Scale to (1, 1, 1). This will cause the image to take up the full screen.

Near the top of the Inspector pane there is a choice box for "Layer", click this and select the "Background" layer you created in step 1.

We need a placeholder texture for the background image. This will be replaced at runtime (with script) by the webcam image. In the asset portion of the Project pane, import a new image as an asset. In the inspector, change the Texture Type to Sprite (2D and UI). Apply the changes when prompted.

Select the BackgroundImage object in the Hierarchy. Under the GUITexture portion of the Inspector, select the recently imported sprite in the Texture properties box.

5. Setup the Main Camera to not clear the background. This is essentially deferring the backbiter to the BackgroundCamera.

Select your "Main Camera" from the Hierarchy tab and open the properties for the "Camera" component. Set the following properties to these values:

Clear Flags = "Depth only" (so the color buffer with the background image isn't cleared).

Culling Mask = Select everything EXCEPT "Background" (i.e. uncheck Background). You should have "Mixed.." where the following are selected: Default TransparentFX, IgnoreRaycast, Water, UI

Depth = 0

6. Test BackBuffer texture.

At this point you should see the background image (from Step 4) with your 3D geometry being drawn in front of it.

We are now ready to replace that texture with the webcam.

7. Adding the Webcamtexture with script.

Select the MainCamera. Add a new script component (C#). Name it DrawWebCam.

NOTE: The script we are attaching here is abitrarily attached to the MainCamera so it will start when the app starts. Nothing in this script has anything to do with the MainCamera rendering

Replace the DrawWebcam script with the following:

public void Start () {
	WebCamTexture webcam = new WebCamTexture();
	GameObject bgimg = GameObject.Find("BackgroundImage");
	bgimg.guiTexture.texture = webcam;
	webcam.Play();
}