UNITY OPENCV BOILERPLATE

INTRODUCTION

Boilerplate project for using OpenCV in Unity.
It allows to create a library written in C++ using OpenCV, that can be imported in Unity.

PREREQUISITES

This project uses the precompiled libraries of OpenCV.
Libraries can also be compiled from the sources, but this is not covered here.

• Download the latest OpenCV libraries from the official website
  At the time of writing, the latest version was "4.1.0":
  https://sourceforge.net/projects/opencvlibrary/files/4.1.0/opencv-4.1.0-vc14_vc15.exe/download
• Extract the downloaded archive

For easier setup and use, an environment variable pointing to the extracted directory can de defined:
E.g.: <OPENCV_INSTALL_DIR>

CREATE THE LIBRARY

The Visual Studio solution in the "UnityOpenCV" directory can be used to generate the library.

The library can also be generated from scratch, as described in the following paragraphs.

REMARK: The following process is using Visual Studio as the IDE. This is thus only valid for Windows.
The steps should however be similar in any IDE or OS.
The version of VS that was used is 2017, though there shouldn't be big differences with other versions of the IDE.

• Create a new empty project
  

  • "File" => "New" => "Project..."
    
  • "Visual C++" => "Empty Project"
    
  • Specify the name of the project (and the solution), as well as its location
    => Any name will do, e.g.: "UnityOpenCV".
    

• Add the boilerplate source files
  

  • Navigate to the directory of the newly created project, and create a subdirectory "source"
    => A different directory structure can be used if desired.
    
  • Copy the 2 source files from the "UnityOpenCV/UnityOpenCV/sources" directory of this repository to the new directory
    
    • "unity_opencv.h"
    • "unity_opencv.cpp" ! - TODO: In order to be used on a different OS, changes in the code are required ("decl")!
      
  • In Visual Studio, add the 2 files to the solution
    
    • "Project" => "Add Existing Item..."
      
    • Navigate to the directory where the 2 files have been copied, and select them.
      

• Set OpenCV include path
  

  • "Project" => "Properties" => "C/C++" => "Additional Include Directories"
    ! - Select "All Configurations" to have settings made for both debug and release configurations
    
  • Add the path to the include directory of the OpenCV installation
    <OPENCV_INSTALL_DIR>\build\include

• Set OpenCV library path
  

  • "Project" => "Properties" => "Linker" => "Additional Library Directories"
    ! - Select "All Configurations" to have settings made for both debug and release configurations
    
  • Add the path to the lib directory of the OpenCV installation
    <OPENCV_INSTALL_DIR>\build\x64\vc15\lib
    (The directory might be different depending on the desired version and build)
    

• Add OpenCV library dependency
  

  • "Project" => "Properties" => "Linker" => "Input" => "Additional Dependencies"
    ! - Do that for each configuration separately (debug & release), adding the specific version of the library:
    
    • Debug: "opencv_world410d.lib"
    • Release: "opencv_world410.lib"

• Set project as a dynamic library
  

  • "Project" => "Properties" => "General" => "Configuration Type"
    ! - Select "All Configurations" to have settings made for both debug and release configurations
    
  • Select "Dynamic Library (.dll)"
    

• Set the desired platform (in my case, 64 bits)
  

  • "Build" => "Configuration Manager" => "Active Solution Platform"
    
  • Select"x64"
    

• Build the (empty) library
  

  • Select the desired configuration (32/64 bits, debug/release)
    
  • "Build" => "Build Solution"
    
  • If everything went OK, a DLL file should have been generated
    

The built library contains an empty method "ProcessImage", and doesn't do anything.

WRITE IMAGE ANALYSIS PROCESS

Some process needs to be added to make the library useful.

=> Implement the "ProcessImage" method.
A lot of documentation and tutorials can be found explaining how to use the OpenCV library.

USE IN UNTIY

• Create a new project or open an existing one
  

• Import the libraries
  

  • Create a "Plugins" directory in "Assets" (if doesn't already exist)
  • Import the OpenCV library:
    
    • "opencv_world410d.dll" for debug use
      
    • "opencv_world410.dll" for release use
      => Copy the file(s) in the "Plugins" directory.
      
  • OpenCV Unity library:
    => Copy the generated library in the "Plugins" directory.
    

• Add the following to a new script or to an existing one:
  

  • Import the "ProcessImage" method from the library:
    

    [DllImport("UnityOpenCV")]
    static extern void ProcessImage(ref byte[] raw, int width, int height);

    Where "UnityOpenCV" is the name of the library.
    It requires to import the Unity Interop Services:
    

    using System.Runtime.InteropServices;

  • Call the method:
    

    ProcessImage(ref rawImage, width, height);

    Where "rawImg" is a byte array:
    

    byte[] rawImage;

Sample Unity Projects

The provided Unity sample projects can be used as example or starting base.

• "unity-opencv": Very basic project acquiring a video feed from the device's webcam and sending the image to OpenCV.
  It is a slightly modified version of Amin Ahmadi's project (see DISCLAIMER below).
  
• "unity-kinect-opencv": Shows how to use OpenCV with the Kinect RGB video feed.
  It is based on the "Green Screen" sample from the MS Kinect SDK.
  

The projects require the OpenCV library as well as the generated C++ library to be added to the Assets in a "plugins" directory.

DISCLAIMER

The project is based on the following posts from Amin Ahmadi:

• "How to Use OpenCV in Unity"
• "How to Pass Images Between OpenCV and Unity"