Profile of a Developer Working with ARToolKit

John Wolf - Developer, ARToolKit Team
March 23, 2016

This is an article about the learning curve to using ARToolKit and creating augmented reality (AR) applications. It includes my experience and my recommendations on how to get up and running with developing AR applications.

At first, I was excited to get involved in AR application development. I happen to think it will be the next killer app. I coin it, “AR Everywhere” because soon AR apps will become ubiquitous. Why? Because of the simple fact that the technology has the potential to turn any inanimate object into a responsive device.

I have a master’s degree in computer science and a bachelors of science in quantitative economic analysis. I have done software development for 25 years. I’ve worked with microcode, big and small embedded systems, mobile devices, and humongous desktop applications. I’ve worked with the programming languages from RISC assembly to WinRT (Windows Phone/Surface Store) C++/CX - clearly the hardest programming language and system APIs I’ve ever had to work with. In my educational background, I had a fair amount of exposure to advanced mathematics and quantitative analysis (however, I never really put them to use in my 25 years of software development).

With all of this, I figured AR application development would be easy. Boy, was I wrong. Below is my journey, which by no means is complete nor will it ever be. I will outline what worked and what didn’t, and present my major takeaways, while leading you through my journey.

Ready, Set, Go, Face Plant…

My first task threw me head long into creating an AR app using the Unity3d UnityPro Editor or integrated development environment (IDE). I was to use the Unity IDE combined with the Qualcomm Vuforia client APIs and services, Unity editor scripting, a Unity player plugin developed using Microsoft Windows 8.1 Phone/Tablet Store APIs and the Windows Visual Studio IDE. I struggled as to where to begin. The four previously mentioned components are by no means light reading topics and all constantly and often significantly changing.

Takeaway #1 - EXPECT CONSTANT AND EXTENSIVE CHANGE. I had to stay connected with resources that kept me apprised of upcoming changes and that would help me continually climb new learning curves.

I soon became aware that AR application development involves many advanced topics:

  • The tools and the process of creating 2D and 3D graphics, or modeling.
  • The tools and the implementation of animation of 2D and 3D graphic models.
  • Instead of coloring, it’s now adding material, texture and shading to 2D and, especially, 3D graphic models. Introduces concepts like ray tracing, UV mapping and ambient occlusion, etc.
  • Implementing graphic applications whether through APIs like OpenGL or gaming engines like Unity3D.
  • Always involves programming to contracts, interfaces, addons or plugins specification. These aren’t necessarily difficult concepts but they are all overloaded terms that differ in meaning and concept from implementation to implementation.
  • The step of adding markers to a target or creating a dataset that represents the target is to be recognized and tracked by a camera device-based application.
  • The many AR algorithms for the step of recognizing moving targets in real-time.
  • The many AR algorithms for the step of tracking a moving target in real-time.
  • The many AR algorithms involving complex mathematics for the step of determining the orientation of the moving target in real-time.
  • The complex AR algorithms that constantly re-render graphic augmentations into the video stream of a viewer device.
  • Writing a constantly looping and event-driven application with a frame rate of 60 frames per second or faster.

Takeaway #2 - DEVELOP A STRATEGY TO STEER A COURSE THROUGH THE MAZE OF LEARNING NEW CONCEPTS. I had to map out an “attack list” of topics to learn, starting at the beginning. My beginning was the beginning: creating a 2D and 3D graphic model.

I had to figure out the most efficient and effective way that I learn and retain. In this fast-moving field, static reference material is quickly outdated. So I strived to find learning resources that dynamically morphed to changing and enhanced technologies. I looked for reference materials that had an evolving index. Or, in the case of static resources, I had to be willing take on the expense of constantly renewing these resources and trashing the outdated ones. The problem with this strategy is that the familiar layout and all margin notes are lost when the outdated material is replaced by the new.

Takeaway #3 - EXTENSIVELY USE AN INDEXED-FOR-SEARCHING NOTE-TAKING APP AND PREFER ONLINE, DYNAMIC, LEARNING RESOURCES. Enough said.

Furthermore, knowing that I’d be working on products developed on Mac OS X, that is, in addition to Windows 10, and that target both Mac OS X and Apple iOS, I had to find a way to have both the Mac OS X and Windows development environments contained in a single, portable, workstation computer. You wouldn’t think this could be possible, but amazingly it is (but that’s another blog entry).

Augmented Reality (AR) Software Development Kit (SDK)

Takeaway #4 - CHOOSE AND USE AN AR SDK!

Most of us can’t write our own AR functionality. There just isn’t enough time in one’s career. So this means you need an AR SDK that implements all the algorithms and services enumerated above. I’m going to recommend the ARToolKit SDK for these great reasons:

  1. ARToolKit has history. ARToolKit is known as the first available AR SDK.
  2. The ARToolKit SDK is free - no royalties for apps that use it.
  3. The great majority of the ARToolKit SDK is open source allowing one to peer into the implementation of AR functionality.
  4. The ARTooKit SDK supports three prominent development platforms: Linux, Mac OS X, and Windows Desktop.
  5. The ARToolkit SDK supports the prominent target app platforms of Android, Mac OS X, Apple iOS, Linux, Windows 8.1/10 Desktop and Windows 8.1/10 Surface/Phone Store.
  6. The ARToolkit SDK supports the graphic APIs of OpenGL and WebGL and the Unity3D game engine.
  7. The ARToolKit SDK contains several sample applications and their source code whose functionality range from simple to complex.
  8. The ARToolKit is alive and well, more so then before. The ARToolKit community is growing.

Now Really: Ready, Set, Go…

I backed off hitting the AR books until I had a basic understanding of 2D and 3D graphic creation and how graphics are colored, lit (virtual lighting) and textured. I had to take time out to get a good understanding of “shaders” and how they are applied to a graphical model.

Then I had to learn how graphics are constructed so that different parts of a graphic model can be positioned independently producing realistic poses and movement. From there, I learned how 2D and 3D graphic models are animated using keyframes and a timeline. However, this was easy for me because of my video and motion graphics editing experience.

Next, I had to look into the concepts of translation, rotation, and scaling as they applied to a graphic model, and how translation, rotation and scaling is tracked inside some singular virtual frame of reference.

With all of that behind me, I was ready to take on AR. Ready to take on target (graphic) recognition, target tracking, calculating target orientation and, finally, augmenting the viewer with 2D or 3D graphic models - animated or static. These all can be implemented using different techniques. Up to this point, I trusted the AR SDK to implement those techniques, and I remain agnostic of the techniques.

That brings me to the big conundrum: How to wrap my head around the inner workings of AR?

Takeaway #5 - START WITH THE SAMPLE AR APPS THAT COME WITH THE AR SDK.

I started this part of the journey by understanding what the simplest sample AR app, which was included with the SDK, source and all, was supposed to do. Then I chose the development platform that, of course, was supported by the SDK, I was most familiar with and the target app platform that, again, was supported by the SDK, I have access to a real device that can be deployed and experimented with (where the built sample app can be executed on). Then, using a debugger, I stepped through an executing app on the device noting where the sample app calls SDK APIs. Next, I looked up the called SDK APIs in the SDK’s documentation to understand what the API is supposed to do and how it’s supposed to be called - while taking notes, of course. Once doing this, the activity was repeated with another sample app until I exhausted all of the sample apps. I started with the simple and proceeded to the complicated.

But I wasn’t done yet! Almost (but not really).

Takeaway #6 - DON’T LIMIT YOURSELF TO A SINGLE TARGET APP PLATFORM.

Learn how to deploy and execute the sample apps to as many target app platforms supported by the SDK as possible. This step may be limited by the expense of acquiring a late model device of the particular target app platform. For example, an Apple iPhone can be pricey.

Takeaway #7 - START THE PROCESS TO UNDERSTAND HOW AR WORKS UNDER THE HOOD.

Learn the terminology. Relearn the long-forgotten math involved. Learn the steps, techniques and algorithms of how AR is coded. Repeat forever.

I have arrived… Not really, because one can only approach understanding the implementation of AR as it races ahead in its innovation, sophistication and complexity.