In our previous article, Intel RealSense D400 librealsense2, we began work on making the the RealSense SDK work more smoothly on the NVIDIA Jetson TX Development kits. We now add CUDA support! Looky here:
Background
As you may recall, there are a few issues with running the RealSense SDK, librealsense 2, on the NVIDIA Jetson TX2. For example, the application “realsense-viewer” pegged a CPU at 100%, which results in poor performance. We also want to add support for the Jetson TX1.
As we had guessed, there was a wild render loop which did not have a sleep call. After fixing that, we add CUDA support. This is a little more of a challenge than one would like. Intel switched over to using CMake in version 2 of librealsense. The first task is to add flags to tell CMake to use CUDA. We chose the appropriately named “USE_CUDA” flag.
Next we add the actual CUDA code itself. Fortunately we have a template to work off. It turns out that Mehran Maghoumi (who is currently working at NVIDIA!) had done some work in his Github repository culibrealsense on adding CUDA support for the image conversion of the native image format (YUY2) to more display friendly RGB. A few nips and tucks later, we now have support for RGB, BGR, RGBA and BGRA in CUDA. BGR is the preferred OpenCV format. The ‘A’ refers to a alpha channel for the image.
There is some speedup using the new conversion code. In a few tests the conversion time of an image went from ~14ms to ~8ms. This is a healthy speedup. The code can probably be improved, but for our purposes it’s a nice speedup for the amount of work involved.
During the development process, we also upgraded to librealsense 2.10.4. The library is under heavy development, unfortunately the challenge is discovering new “features” which don’t translate well into the Jetson environment. These tend to be things such as code written for Intel x86 processors and the myriad of idiocy that goes along with multi-platform libraries. To be fair, this is an Intel product. It makes sense that there are optimizations for their processors. It just takes a while to figure out what breaks when the optimizations are added.
The kernel patching routines are rewritten to support the Jetson TX1, and to be slightly more intelligent about configuring the kernel.
Note: Because space is tight on the Jetson TX1, you should first flash the Jetson TX1 with L4T 28.2 using JetPack 3.2, download the buildLibrealsense2TX repository, and then build the patched kernel. Once the kernel is in place, remove the kernel sources to give you enough space to operate.
Installation
On the JetsonHacks Github account, there is a repository buildLibrealsense2TX. To download the repository:
$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
$ git checkout v0.8
The instructions are much the same as in the previous article. More details are available on the README.md file in the buildLibrealsense2TX directory.
Explain please
First, a word about what we’re doing. There are several RealSense camera formats that the standard UVC (video) module in the kernel does not support. librealsense provides patches to add those video modes to the appropriately. The patches also add support for properly adding timestamps to the incoming video stream. In addition, there are several patches that modify some of the Industrial I/O (IIO) tree. These patches are providing support to supplemental hardware on a RealSense camera, such as a 3D Gyro or 3D Accelerometer.
Note: Some of the patches apply to modules that are built into the kernel itself. Because these modules are required to be in the Image, and not built as external, you will need to compile the kernel itself along with the modules.
If you do not modify the Linux kernel, librealsense will mostly appear to work. You will probably experience issues a little further along when you are trying to get better precision and accuracy from the video and depth streams. As the RealSense D435 camera does not have supplemental hardware, the IIO patches don’t make any difference.
If you’re just looking to play around with the camera, you may be able to get away with not compiling the kernel and skip over it. If you’re more serious, you’ll have to start patchin’ !
Building the Kernel
Note: If you built your kernel as in the previous article, you must rebuild it again!
Building the kernel can be taxing, there are many little things that can go wrong. Plus, you can make your Jetson become brickly if something happens at an untoward momement. If you’re a go for it kind of person:
$ ./buildPatchedKernelTX2.sh
If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.
$ ./buildPatchedKernelTX2.sh –nocleanup
Actually, this script is more useful as a template for rebuilding your own kernel with the librealsense changes. There are two scripts in the ‘scripts’ directory for helping in the kernel build:
patchKernel.sh applies the librealsense kernel patches.
configureKernel.sh Configures the kernel to add the appropriate modules needed by librealsense.
Since you’re a developer, you should be able to figure out what the scripts are doing and modify them to match your needs.
librealsense Installation
make sure that the RealSense camera is not attached to the system. Then install the library:
$ ./installLibrealsense.sh
Looking inside the script file, you will see that there are a couple of patches for the Jetson. The first patch is a work around for some code related to an Intel specific instruction set, the other is a workaround for the Industrial I/O (IIO) device detection.
The stock librealsense code appears to assume that a IIO device reports with a device and bus number. On the Jetson, the ina3221x Power Monitors do not follow this protocol. The result is that a series of warning are issued continuously as the library scans for HID devices that have been added (plugged in) to the system.
The library is looking for IIO HID devices (the accelerometer and gyroscope on a RealSense camera). The ina3221x is not a HID device, but appears during the IIO scanning. The library scans, but because it does not find a device or bus number for the power monitor, it issues a warning to stderr (the console). The result is that the console gets spammed, which in turn results in a performance penalty.
The workaround patch checks to see if the device detected is the power monitor before issuing a warning.
Similarly, the built in camera module control is via CSI/I2C, not USB as expected by librealsense. Again, a warning is sent to stderr by librealsense. There may be a clever way to determine if the warning is about the on-board Jetson camera, but in this case the patch just comments out the warning.
After applying the patches, the script compiles the library, examples and tools:
- The library is installed in /usr/local/lib
- The header files are in /usr/local/include
- The examples and tools are located in /usr/local/bin
The script also sets up a udev rule so that the RealSense camera is available in user space.
Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:
$ cd /usr/local/bin
$ ./realsense-viewer
Conclusion
So there you have it. This is the first pass through of getting the RealSense D400 cameras working with the Jetson TX Dev kits. Because the RealSense SDK is under heavy development, we will have to keep our eye out for improvements in the weeks ahead!
Notes
- L4T 28.2 installed by JetPack 3.2
- A Jetson TX2 was used in the video
- buildLibrealsense2TX is version v0.8
Intel Documentation:
30 Responses
Hello-
YOur videos/instructions are clear and concise enough for even a newbie like myself.
I was wondering if anyone has tried using the RS D-400 cameras with a smartphone.
Whattya think?
Thanks,
TM
That sounds challenging. Thanks for reading!
AWESOME! Hi Jim, do you think the D435 is a candidate for replacing / augmenting the ZED or structure.io cameras on RACECAR/J?
I’m also curious about this… Anyone care to comment on ZED vs RealSense for indoor SLAM?
Hello Jim,
Thank you for your tutorials. It help me so much! I have followed the previous tuturial to install realsense D415 on my Jetson TX2 and patch the kernel.
But I have just discovered this new tutorial, I’m trying to follow it, but I have an issue during the installLibrealsense process. It takes only a few second and I get the following error:
CMake Error at /usr/share/cmake-3.5/Modules/FindCUDA.cmake:617 (message):
Specify CUDA_TOOLKIT_ROOT_DIR
Call Stack (most recent call first):
CMakeLists.txt:61 (find_package)
and the process stop.
Then, if I run the realsense viewer after running the nvpmodel command I get a CPU to 100% without running the camera…
Could you help me with this issue, please?
Thanks
Rich
What version of L4T are you running? Do you actually have CUDA installed on the device?
FindCUDA could not find the CUDA toolkit the error message says.
Hi Jim,
I finally solved my issue by a new complete install and now all is ok.
Thanks for your help.
Rich
You’re welcome, and I’m glad you got it to work!
I’m using L4T 28.2 and Cuda was normally installed with the Jetpack 3.2, but you are right, it seems that CUDA was not install during the process.
I will investigate on that way and let I will you know if I found the solution.
Thanks for your quick answer
Hi Jim,
Thanks for your tutorials. Were you able to build the real sense library for Python wrapper?
You are welcome! I did not try to build the Python wrappers. Thanks for reading!
Thank you for an informative and fair review.
We looked into some of the problems reported here and integrated number of fixes into our latest release (including BUILD_WITH_CUDA CMake flag – https://github.com/IntelRealSense/librealsense/pull/1866)
Hopefully, this will improve the experience for our Jetson users.
Intel RealSense team
Hi Sergey,
Thank you for taking the time to add the changes to your library. I know that the Jetson is not an officially supported platform, and appreciate the effort to add support.
Hi,
Is there anybody tested the glass-to-glass latency of this camera on TX2 ? Thank you very much !
Hi,
I am having a really hard time to install this library. At the end of the installation process I get the following error message: (I dont have CUDA installed because Im having trouble there too. Other than that I have Jetpack 3.2.1 and L4T 28.2 installed on my Jetson tx2)
— Info: REALSENSE_VERSION_STRING=2.10.4
CMake Error at /home/nvidia/CMake/Modules/FindCUDA.cmake:687 (message):
Specify CUDA_TOOLKIT_ROOT_DIR
Call Stack (most recent call first):
CMakeLists.txt:61 (find_package)
— Configuring incomplete, errors occurred!
See also “/home/nvidia/librealsense/build/CMakeFiles/CMakeOutput.log”.
Building librealsense, headers, tools and demos
make: *** No targets specified and no makefile found. Stop.
Please examine the script files and modify them to suit your application. There is a flag you can turn off which does not include CUDA. However, you should fix you CUDA installation.
Thanks for your tutorials. I have installed L4T 28.1, CUDA 8.0 and ROS on a Jetson TX2 according to the article “RACECAR/J Software Install” for my RACECAR/J. I have also programmed my FOCBOX VESC and tested teleoperation with the F710 gamepad. Would I be able to use the installation procedure in this article to run the Intel RealSense D435i camera with my RACECAR/J with its current software stack? The note at the end of this article refers to L4T 28.2. If not how can I run the D435i with L4T 28.1?
You will need to update the scripts to use the kernel for L4T 28.2. Thanks for reading!
I installed L4T 28.2 on my Jetson TX2 and ran installLibrealsense.sh (without
the D435i camera connected) and buildPatchedKernel.sh scripts. When I run rs-enumerate-devices I get the response “No device detected. Is it plugged in?”
and if I run realsense-viewer, my camera is not connected. lsusb shows
“Bus 002 Device 003: ID 8086:0b3a Intel Corp.”, modinfo uvcvideo | grep “version:” shows
version: 1.1.1
srcversion: 5DDB0A26F9B02446957845B
and dmesg shows:
[ 9.756108] usb 2-1.4: New USB device found, idVendor=8086, idProduct=0b3a
[ 9.764836] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 9.764838] usb 2-1.4: Product: Intel(R) RealSense(TM) Depth Camera 435i
[ 9.764840] usb 2-1.4: Manufacturer: Intel(R) RealSense(TM) Depth Camera 435i
[ 9.764842] usb 2-1.4: SerialNumber: xxxxxxxxxxxx (a 12 digit number)
.
.
.
[ 10.324607] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435i (8086:0b3a)
[ 10.347508] uvcvideo: Unable to create debugfs 2-3 directory.
[ 10.355655] input: Intel(R) RealSense(TM) Depth Ca as /devices/3530000.xhci/usb2/2-1/2-1.4/2-1.4:1.0/input/input8
[ 10.368289] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435i (8086:0b3a)
[ 10.383966] uvcvideo: Unable to create debugfs 2-3 directory.
[ 10.392595] usbcore: registered new interface driver uvcvideo
[ 10.400855] USB Video Class driver (1.1.1)
Any suggestions on how to proceed to get the D435i camera working on the Jetson TX2?
This article installs version 2.13 of librealsense. You can try moving up to a more recent release to see if that helps.
Version 2.18 solved the main issue. I am able to use the Stereo Module, RGB Camera and Motion Module.
I do get the following error on realsense-viewer startup and on toggling the
Stereo Module:
“Invalid Value in rs2_get_option(options:0x7f8400d130,option:Emitter On Off):
hwmon command 0x7d failed.Error type: Invalid Command (-1)”
and a string of “Incomplete video frame detected!” messages on turning on
the RGB camera (1280×720, 30 FPS, RGB8). I have to go down to 848×480 resolution
to avoid these messages and at 60 FPS even this lower resolution will give
the above error message.
Any chance that this will be updated to support Xavier 4.1.1?
Thanks!
I have the issue:
No CMAKE_CUDA_COMPILER could be found
The solution to this problem can be found in the following link:
https://github.com/jetsonhacks/buildLibrealsense2TX/issues/13
Basically, I have just flashed my TX2, and the following is the content of the file /etc/environment (I left path unchanged, and added the second line to it):
PATH=”/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games”
CUDACXX=/usr/local/cuda-9.0/bin/nvcc
Hi
I’m having an issue during building of the kernel. The “Image” folder in /usr/src/kernel/kernel-4.4/arch/arm64/boot is not present after the “makeKernel.sh” script has finished. There is an Image folder in /usr/src/kernel/kernel-4.4/arch/mips/lasat/image that contains these files: Makefile, head.S and romscript.normal. I’m guessing it’s not the same?
Do you have any idea what is wrong? The makeKernel.sh finished nicely with the last output line “DEPMOD 4.4.38-tegra” just like in your youtube video
You will need to provide more information:
Which Jetson are you using?
Which version of L4T is installed?
The depmod is the last information printed after a long build. It does not indicate anything other than that the modules were updated. The kernel Image is created in the first part of the build, long before depmod is called.
BOARD TX2, ubuntu 16.04, kernel 4.4.38, L4T 28.2.1
system on SSD
Hello,
can’t install realsense2_ros kinetic package, !!
message : librealsense2 is missing ?!!
kernel build and librealsense2 2.10.2 process ok
…
— Install configuration: “Release”
— Installing: /usr/local/lib/librealsense2.so.2.10.2
— Installing: /usr/local/lib/librealsense2.so.2
— Up-to-date: /usr/local/lib/librealsense2.so
— Up-to-date: /usr/local/include/librealsense2
— Up-to-date: /usr/local/include/librealsense2/h
…
tested with this link : https://github.com/IntelRealSense/librealsense/blob/master/doc/installation.md
no errors.
tested with the present sh :
./buildPatchedKernelTX2.sh
SSD/Boot/image copied to MMC/Boot/image
reboot and ./installLibrealsense.sh ok
I think that the librealsense2 is too old !!
do you know how to compile the 2.19.0 or + version ?
Thanks friends
Vincent
Check line 10 of the “installLibrealsense.sh” script with “LIBREALSENSE_VERSION=v2.13.0”
Maybe you can change this to your preferred version
Eirik, THANKS THANKS THANKS !!!!
You helped me a lot !!!
I had success with 2.19.2 lib on realsense2-camera ROS Kinetic !!!
I’m really really happy !!!
My process, for others :
– git clone librealsense_master to $home,
– changed line 10 of the “installLibrealsense.sh” script with “LIBREALSENSE_VERSION=v2.19.2” (the last visible tag on librealsense github)
– ./installLibrealsense.sh
SUCCESS !!!
Just curious, does cuda support also availbale on jetson xavier?
I believe so.