JetsonHacks

Developing on NVIDIA® Jetson™ for AI on the Edge

ROS, OpenCV and OpenCV4Tegra on the NVIDIA Jetson TK1

This is a little opinion piece on running Robot Operating System (ROS) with OpenCV versus OpenCV4Tegra on the NVIDIA Jetson TK1. First let’s talk about some of the advantages that OpenCV4Tegra has versus regular OpenCV.

As you can read from Jetson/Installing OpenCV, OpenCV4Tegra is a CPU and GPU accelerated version of the standard OpenCV library. OpenCV stands for “Open Computer Vision”, the de-facto standard Computer Vision library containing more than 2500 computer vision & image processing & machine learning algorithms. You can read more about the specific performance benefits of using OpenCV4Tegra versus OpenCV at Jetson/Computer Vision Performance.

Note: As of this writing, this is for the current 2.4.10 release of OpenCV vs. 2.4.10 release of OpenCV4Tegra.

Background

There are three versions of OpenCV that you can run on the Jetson:

  1. “Regular” OpenCV
  2. OpenCV with GPU enhancements
  3. OpenCV4Tegra with both CPU and GPU enhancements

“Regular” OpenCV is OpenCV that is compiled from the OpenCV repository, with no hardware acceleration. This is typically not used on the Jetson, as GPU enhancements are available for OpenCV. OpenCV with GPU enhancements is designed for CUDA GPGPU acceleration. This is part of the standard OpenCV package. OpenCV4Tegra is a free, closed source library available from NVIDIA which includes ARM NEON SIMD optimizations, multi-core CPU optimizations and some GLSL GPU optimizations.

So why wouldn’t you always use OpenCV4Tegra? The answer lies in the actual OpenCV library itself; there are two proprietary patented algorithms, SIFT and SURF, which exist in opencv-nonfree. Because these are patented, NVIDIA does not include them in their distribution of OpenCV4Tegra. Therefore if your code does not use SIFT or SURF, then you can use OpenCV4Tegra and get the best performance.

So why use SIFT and/or SURF? The quick answer is that when people are doing feature detection, SIFT/SURF are two of the most popular algorithms in use today. One application is simultaneous Localization And Mapping (SLAM) used mostly in the robotics/drone world. One of the most popular packages of which is Semi-Direct Monocular Visual Odometry (SVO), available on Github at rpg_svo.

Another application which uses SIFT/SURF is deep learning, such as the package Caffe Deep Learning Framework.

Alternatives?

The first alternative is that if you do not have any need of SIFT/SURF in your application, you can use OpenCV4Tegra and enjoy the best performance. There is a rub, but also a possible workaround.

When OpenCV4Tegra is installed, the packaging is different from what most packages, such as Robot Operating System (ROS), expect. Aptitude, the Debian package manager, reports that when OpenCV4Tegra is installed, libopencv-* is not installed. Aptitude notices a conflict with ‘upstream opencv’ (in this case regular old opencv) and requests that OpenCV4Tegra be uninstalled before OpenCV is installed. Jetson forum member davywb figured out a work around, and posted it to the Jetson forum. While the workaround is for ROS, it will probably work for most other packages too.

If you need SIFT/SURF you can:

  • Use OpenCV4Tegra, analyze the public OpenCV source code, and then copy/paste the parts of the nonfree module that you want (eg: SURF feature detector) from OpenCV into your own project. You will have the CPU optimizations of OpenCV4Tegra for most of your code and will have the GPU module and will have the non-optimized patented code that you need from the nonfree package such as SURF. So this option gives full performance (for everything except the nonfree code) but is tedious (and difficult to maintain).
  • Ignore OpenCV4Tegra, and instead, download & build public OpenCV. You will still have the GPU module but not any CPU optimizations, but you won’t need to spend time ripping out parts of the OpenCV non-free module code. So this option is easiest but produces slower code if you are running most of your code on CPU.
  • You can read more at the Jetson/Installing OpenCV page.

    Opinion

    If you need SIFT/SURF, then you should just build OpenCV from source, otherwise use OpenCV4Tegra.

    Notes

    Note 1: I would only use OpenCV4Tegra with the added SIFT/SURF code from the OpenCV library as a horrible last resort to wring out last bit of possible performance from the little beast. The major issue with the combination is that it’s a maintenance nightmare; you have to track all the different versions of the OpenCV library and OpenCV4Tegra and merge them. As you know, maintenance is the real bill that you have to pay when writing code.

    The mistake that most people make is premature optimization. Instead of building something that works and then fine tuning it, programmers tend to spend too much time working on optimizing code which may not be a bottle neck at any time in the near future. Build something that works, figure out where the time is being spent, and then optimize. The closer to the metal that you are (GPU and CPU optimizations), the more time consuming (and error prone) the project becomes. If you get something to work, you can most likely speed it up. If you can’t get to the point where it works at all, you are working on optimizing a failure path. It’s the old 80/20 rule, put the 80% in the bank first and see where you really get the most benefit from the last 20% and work on that.

    Note 2: OpenCV 3.0 handles SIFT/SURF in a separate repository, opencv_contrib repo. This may make it easier in the future to combine OpenCV4Tegra with SIFT/SURF, but because OpenCV4Tegra is still at release 2.4.10, this remains to be seen.

    Facebook
    Twitter
    LinkedIn
    Reddit
    Email
    Print

    8 Responses

    1. 1. Is there a way to know Opencv4Tegra be using in application
      maybe just use standard OpenCV instead of Opencv4Tegra

      2.While use Opencv4Tegra, do both of them different in code like different function, lib ..

      Thank you.

      1. Hi SeanYao,

        1. Is there a way to know Opencv4Tegra be using in application
        maybe just use standard OpenCV instead of Opencv4Tegra

        Most applications that are built do not use OpenCV4Tegra. The OpenCV4Tegra library has to be built into your executable, which usually means that it is purpose built. If you are writing your own programs, for example, you can use OpenCV4Tegra.

        2.While use Opencv4Tegra, do both of them different in code like different function, lib ..

        Except for the “non-free” libraries, the function calls are the same. The libraries, of course, are different. One is the OpenCV library that you would compile (or install using apt-get), the other is the OpenCV4Tegra.

    2. You state that deep learning frameworks, such as caffe, use SIFT/SURF features. I do not have direct experience with this framework, but I am familiar with the theoretical stuff about Deep Learning.

      I am planning to use a Jetson TK1 for deep learning, and use Caffe. But I would also like the speed from OpenCV4Tegra. Can you shed some light on the subject of how Caffe uses SIFT/SURF?

      1. It’s beyond the scope of a blog post to describe the process, but basically SIFT/SURF algorithms are local feature detectors and descriptors that are used for object recognition, registration, or 3D reconstruction. Here’s the simple version of what the algorithms do:
        https://en.wikipedia.org/wiki/Speeded_up_robust_features
        You can also find the information on the OpenCV site. In a nutshell, “deep learning” takes this information (which is basically a pyramid) and applies it across a large number of images which (hopefully) to provide a more canonical, generic pyramid. A trained feature set, if you will.
        More info about installing Caffe on the Jetson TK1 here:
        https://jetsonhacks.com/2015/01/17/nvidia-jetson-tk1-caffe-deep-learning-framework/

    3. If you have used Jetpack to install Opencv4Tegra and such, how can you back it all out and just go on with OpenCV ?

      1. I think it responds to the usual persuasion, depending on how brave you are. If you use ‘Synaptic Pacakage Manager’, run quick filter opencv4tegra and you should see libopencv4tegra-repo, libopencv4tegra, and libopencv4tegra-dev selected and installed. You can use Synaptic to remove them. You can also do it through the command line if you’re a little more adventurous:

        $ sudo apt-get purge libopencv4tegra libopencv4tegra-dev libopencv4tegra-repo

        I’ve never tried it this way, I have always just done a fresh install without opencv4tegra when I wanted to use OpenCV by itself.

        Hope this helps.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Disclaimer

    Some links here are affiliate links. If you purchase through these links I will receive a small commission at no additional cost to you. As an Amazon Associate, I earn from qualifying purchases.

    Books, Ideas & Other Curiosities