Integrating OpenCV with ARToolkit
OpenCV is a powerful open source Computer Vision library from Intel which provides a wide range of tools.
- General Image Processing
- Object Matching and Recognition
- Machine Learning
Developing in Microsoft Visual Studio, Opencv is in fact cross platform and this code will work for all variants.
In this tutorial we are going to preprocess the video in ARToolkit to deal with poor lighting situations using OpenCV’s Retina Filter. Improvements in lighting can help the ARToolkit tracker detect and follow markers in poor lighting conditions.
Open the Visual Studio solution for ARToolkit 5. Select the project called SimpleLite, this demo uses a simple square marker and draws a cube using GLUT.
To avoid redundant code, ARToolkit comes with only a selection of OpenCV modules. In order to use the wider functionality you must get OpenCV from the dowload page, this tutorial is using version 2.4.10 which is the same version used by ARToolkit 5.
Install OpenCV using the installer or consule the readme for your system.
Copy the newly installed libs from (opencv\build\x86\vc12\lib) to the appropriate ARToolkit Project folder (\lib\win32-i386 or \lib\win64-x64) overwritting the existing libs files listed.
Copy the newly installed DLLs from (opencv\build\x86\vc12\bin) to the appropriate ARToolkit Project folder (ARToolkit5\bin) overwritting the existing libs files listed.
Configure Visual Studio
Go to Project Properties -> Linker -> Input
Additional Dependancies add “opencv_core2410.lib;opencv_calib3d2410.lib;opencv_imgproc2410.lib;opencv_highgui2410.lib;opencv_nonfree.lib;”
OpenCV is a C++ library, so make sure the your project is configure to compile C++.
Project Properties -> C/C++ -> Advanced -> Compile As
#include "opencv2\opencv.hpp" #include "opencv2\opencv_modules.hpp"
Create Global Variables
static cv::Retina *retina = NULL; static cv::Mat imgOpenCV; //SMART POINTERS!!!!
#### PLEASE NOTE: The type cv::Mat is a Smart Pointer type by design, do not use POINTERS when handling it.
With the global variables created, they need to be instantiated in the setupCamera() method once at the beginning of the program.
Create the image from the determined camera size and the same with the retina.
imgOpenCV = cv::Mat(xsize, ysize, CV_8UC4); //Note RGBA format for ARToolkit images retina = new cv::Retina(cv::Size(xsize, ysize));
Convert ARImage* to cv::Mat
The ARImage type is updated on each pass of the GLUT main loop.
Since the imgOpenCV cv::Mat exists and is of the correct size, it is possible to cast the data from the ARImage to the cv::Mat data property. Using OpenCV’s highgui lib, the image can be displayed to show that is is contained within the cv::Mat correctly.
imgOpenCV.data = (uchar *)image; cv::imshow("OpenCV Image", imgOpenCV); //Disp[lay the image in an OpenCV window for debug purposes cv::waitKey(); //Press return on OpenCV window
A function is now applied to the cv::Mat which should improve any lighting issues within the image, potentially removing some shadows from the scene.
imgOpenCV.data = (uchar *)image; cv::Mat imgIn; //3 Channel temp image cv::cvtColor(imgOpenCV, imgIn, CV_BGRA2BGR); retina->run(imgOpenCV); cv::Mat retinaOutput_parvo; retina->getParvo(retinaOutput_parvo); //Show the OpenCV image for debug purposes. cv::imshow("OpenCV Image", retinaOutput_parvo); cv::waitKey(); //Convert back to BGRA when finished processing. cv::cvtColor(retinaOutput_parvo, imgOpenCV, CV_BGR2BGRA);
The retina function returns two matrices, the parvo cellular and magno cellular. For lighting correction, it is the parvocellar image that is of use in further processing.
Convert cv::Mat to ARImage*
It is possible to pass the OpenCV image back to the ARImage, be sure to have the gARImage altered to run the ARTracker on the processed image and for it to be displayed.
image = retinaOutput_parvo.data; gARImage = retinaOutput_parvo.data;
Although OpenCV uses Smart Pointers that do some degree of garbage collection it is still good practice to release the OpenCV objects in the cleanUp() method of SimpleLite.c
The release method will clear the data, as at this stage, these are the only pointers to the underlying image objects.
This is just one example of the possibilities of combining OpenCV and ARToolkit.