Integrating OpenCV with ARToolkit

Dan Bell
May 23, 2016

OpenCV is a powerful open source Computer Vision library from Intel which provides a wide range of tools.

  • General Image Processing
  • Object Matching and Recognition
  • Machine Learning

Developing in Microsoft Visual Studio, Opencv is in fact cross platform and this code will work for all variants.

In this tutorial we are going to preprocess the video in ARToolkit to deal with poor lighting situations using OpenCV’s Retina Filter. Improvements in lighting can help the ARToolkit tracker detect and follow markers in poor lighting conditions.

after using OpenCV

SimpleLite.c

Open the Visual Studio solution for ARToolkit 5. Select the project called SimpleLite, this demo uses a simple square marker and draws a cube using GLUT.

Install OpenCV

To avoid redundant code, ARToolkit comes with only a selection of OpenCV modules. In order to use the wider functionality you must get OpenCV from the dowload page, this tutorial is using version 2.4.10 which is the same version used by ARToolkit 5.

Install OpenCV using the installer or consule the readme for your system.

Copy the newly installed libs from (opencv\build\x86\vc12\lib) to the appropriate ARToolkit Project folder (\lib\win32-i386 or \lib\win64-x64) overwritting the existing libs files listed.

Copy the newly installed DLLs from (opencv\build\x86\vc12\bin) to the appropriate ARToolkit Project folder (ARToolkit5\bin) overwritting the existing libs files listed.

Configure Visual Studio

Go to Project Properties -> Linker -> Input

Additional Dependancies add “opencv_core2410.lib;opencv_calib3d2410.lib;opencv_imgproc2410.lib;opencv_highgui2410.lib;opencv_nonfree.lib;”

OpenCV is a C++ library, so make sure the your project is configure to compile C++.

Project Properties -> C/C++ -> Advanced -> Compile As

Create Includes

#include "opencv2\opencv.hpp"
#include "opencv2\opencv_modules.hpp"

Create Global Variables

static cv::Retina *retina = NULL;
static cv::Mat imgOpenCV;  //SMART POINTERS!!!!

#### PLEASE NOTE: The type cv::Mat is a Smart Pointer type by design, do not use POINTERS when handling it.

With the global variables created, they need to be instantiated in the setupCamera() method once at the beginning of the program.

setupCamera()

Create the image from the determined camera size and the same with the retina.

imgOpenCV = cv::Mat(xsize, ysize, CV_8UC4); //Note RGBA format for ARToolkit images
retina = new cv::Retina(cv::Size(xsize, ysize));

Convert ARImage* to cv::Mat

The ARImage type is updated on each pass of the GLUT main loop.

mainLoop()

Since the imgOpenCV cv::Mat exists and is of the correct size, it is possible to cast the data from the ARImage to the cv::Mat data property. Using OpenCV’s highgui lib, the image can be displayed to show that is is contained within the cv::Mat correctly.

imgOpenCV.data = (uchar *)image;
cv::imshow("OpenCV Image", imgOpenCV); //Disp[lay the image in an OpenCV window for debug purposes
cv::waitKey(); //Press return on OpenCV window

A function is now applied to the cv::Mat which should improve any lighting issues within the image, potentially removing some shadows from the scene.

imgOpenCV.data = (uchar *)image;

cv::Mat imgIn;                                  //3 Channel temp image
cv::cvtColor(imgOpenCV, imgIn, CV_BGRA2BGR);

retina->run(imgOpenCV);
cv::Mat retinaOutput_parvo;
retina->getParvo(retinaOutput_parvo);

//Show the OpenCV image for debug purposes.
cv::imshow("OpenCV Image", retinaOutput_parvo);
cv::waitKey();

//Convert back to BGRA when finished processing.
cv::cvtColor(retinaOutput_parvo, imgOpenCV, CV_BGR2BGRA);

The retina function returns two matrices, the parvo cellular and magno cellular. For lighting correction, it is the parvocellar image that is of use in further processing.

parvo cellular magno cellular

Convert cv::Mat to ARImage*

It is possible to pass the OpenCV image back to the ARImage, be sure to have the gARImage altered to run the ARTracker on the processed image and for it to be displayed.

image = retinaOutput_parvo.data;
gARImage = retinaOutput_parvo.data;

Clean Up

Although OpenCV uses Smart Pointers that do some degree of garbage collection it is still good practice to release the OpenCV objects in the cleanUp() method of SimpleLite.c

cleanup()

The release method will clear the data, as at this stage, these are the only pointers to the underlying image objects.

imgOpenCV.release();
retina.clearBuffers();

This is just one example of the possibilities of combining OpenCV and ARToolkit.