This is a little opinion piece on running Robot Operating System (ROS) with OpenCV versus OpenCV4Tegra on the NVIDIA Jetson TK1. First let’s talk about some of the advantages that OpenCV4Tegra has versus regular OpenCV.
As you can read from Jetson/Installing OpenCV, OpenCV4Tegra is a CPU and GPU accelerated version of the standard OpenCV library. OpenCV stands for “Open Computer Vision”, the de-facto standard Computer Vision library containing more than 2500 computer vision & image processing & machine learning algorithms. You can read more about the specific performance benefits of using OpenCV4Tegra versus OpenCV at Jetson/Computer Vision Performance.
Note: As of this writing, this is for the current 2.4.10 release of OpenCV vs. 2.4.10 release of OpenCV4Tegra.
Background
There are three versions of OpenCV that you can run on the Jetson:
- “Regular” OpenCV
- OpenCV with GPU enhancements
- OpenCV4Tegra with both CPU and GPU enhancements
“Regular” OpenCV is OpenCV that is compiled from the OpenCV repository, with no hardware acceleration. This is typically not used on the Jetson, as GPU enhancements are available for OpenCV. OpenCV with GPU enhancements is designed for CUDA GPGPU acceleration. This is part of the standard OpenCV package. OpenCV4Tegra is a free, closed source library available from NVIDIA which includes ARM NEON SIMD optimizations, multi-core CPU optimizations and some GLSL GPU optimizations.
So why wouldn’t you always use OpenCV4Tegra? The answer lies in the actual OpenCV library itself; there are two proprietary patented algorithms, SIFT and SURF, which exist in opencv-nonfree. Because these are patented, NVIDIA does not include them in their distribution of OpenCV4Tegra. Therefore if your code does not use SIFT or SURF, then you can use OpenCV4Tegra and get the best performance.
So why use SIFT and/or SURF? The quick answer is that when people are doing feature detection, SIFT/SURF are two of the most popular algorithms in use today. One application is simultaneous Localization And Mapping (SLAM) used mostly in the robotics/drone world. One of the most popular packages of which is Semi-Direct Monocular Visual Odometry (SVO), available on Github at rpg_svo.
Another application which uses SIFT/SURF is deep learning, such as the package Caffe Deep Learning Framework.
Alternatives?
The first alternative is that if you do not have any need of SIFT/SURF in your application, you can use OpenCV4Tegra and enjoy the best performance. There is a rub, but also a possible workaround.
When OpenCV4Tegra is installed, the packaging is different from what most packages, such as Robot Operating System (ROS), expect. Aptitude, the Debian package manager, reports that when OpenCV4Tegra is installed, libopencv-* is not installed. Aptitude notices a conflict with ‘upstream opencv’ (in this case regular old opencv) and requests that OpenCV4Tegra be uninstalled before OpenCV is installed. Jetson forum member davywb figured out a work around, and posted it to the Jetson forum. While the workaround is for ROS, it will probably work for most other packages too.
If you need SIFT/SURF you can:
You can read more at the Jetson/Installing OpenCV page.
Opinion
If you need SIFT/SURF, then you should just build OpenCV from source, otherwise use OpenCV4Tegra.
Notes
Note 1: I would only use OpenCV4Tegra with the added SIFT/SURF code from the OpenCV library as a horrible last resort to wring out last bit of possible performance from the little beast. The major issue with the combination is that it’s a maintenance nightmare; you have to track all the different versions of the OpenCV library and OpenCV4Tegra and merge them. As you know, maintenance is the real bill that you have to pay when writing code.
The mistake that most people make is premature optimization. Instead of building something that works and then fine tuning it, programmers tend to spend too much time working on optimizing code which may not be a bottle neck at any time in the near future. Build something that works, figure out where the time is being spent, and then optimize. The closer to the metal that you are (GPU and CPU optimizations), the more time consuming (and error prone) the project becomes. If you get something to work, you can most likely speed it up. If you can’t get to the point where it works at all, you are working on optimizing a failure path. It’s the old 80/20 rule, put the 80% in the bank first and see where you really get the most benefit from the last 20% and work on that.
Note 2: OpenCV 3.0 handles SIFT/SURF in a separate repository, opencv_contrib repo. This may make it easier in the future to combine OpenCV4Tegra with SIFT/SURF, but because OpenCV4Tegra is still at release 2.4.10, this remains to be seen.
8 Responses
1. Is there a way to know Opencv4Tegra be using in application
maybe just use standard OpenCV instead of Opencv4Tegra
2.While use Opencv4Tegra, do both of them different in code like different function, lib ..
Thank you.
Hi SeanYao,
1. Is there a way to know Opencv4Tegra be using in application
maybe just use standard OpenCV instead of Opencv4Tegra
Most applications that are built do not use OpenCV4Tegra. The OpenCV4Tegra library has to be built into your executable, which usually means that it is purpose built. If you are writing your own programs, for example, you can use OpenCV4Tegra.
2.While use Opencv4Tegra, do both of them different in code like different function, lib ..
Except for the “non-free” libraries, the function calls are the same. The libraries, of course, are different. One is the OpenCV library that you would compile (or install using apt-get), the other is the OpenCV4Tegra.
Thank you very much
You state that deep learning frameworks, such as caffe, use SIFT/SURF features. I do not have direct experience with this framework, but I am familiar with the theoretical stuff about Deep Learning.
I am planning to use a Jetson TK1 for deep learning, and use Caffe. But I would also like the speed from OpenCV4Tegra. Can you shed some light on the subject of how Caffe uses SIFT/SURF?
It’s beyond the scope of a blog post to describe the process, but basically SIFT/SURF algorithms are local feature detectors and descriptors that are used for object recognition, registration, or 3D reconstruction. Here’s the simple version of what the algorithms do:
https://en.wikipedia.org/wiki/Speeded_up_robust_features
You can also find the information on the OpenCV site. In a nutshell, “deep learning” takes this information (which is basically a pyramid) and applies it across a large number of images which (hopefully) to provide a more canonical, generic pyramid. A trained feature set, if you will.
More info about installing Caffe on the Jetson TK1 here:
https://jetsonhacks.com/2015/01/17/nvidia-jetson-tk1-caffe-deep-learning-framework/
If you have used Jetpack to install Opencv4Tegra and such, how can you back it all out and just go on with OpenCV ?
I think it responds to the usual persuasion, depending on how brave you are. If you use ‘Synaptic Pacakage Manager’, run quick filter opencv4tegra and you should see libopencv4tegra-repo, libopencv4tegra, and libopencv4tegra-dev selected and installed. You can use Synaptic to remove them. You can also do it through the command line if you’re a little more adventurous:
$ sudo apt-get purge libopencv4tegra libopencv4tegra-dev libopencv4tegra-repo
I’ve never tried it this way, I have always just done a fresh install without opencv4tegra when I wanted to use OpenCV by itself.
Hope this helps.