Recently the Intel Librealsense Development Team added CUDA support to the Librealsense SDK. Here’s how to install it on the Jetson TX Dev kits. Looky here:
Background
As you may recall, we added CUDA support to librealsense in a previous article, Now with CUDA! Intel RealSense D400 cameras – NVIDIA Jetson TX. Recently I received a nice note from the Librealsense Development Team stating that they had used the code as a basis for adding CUDA support to librealsense (starting in version 2.13.0).
Intel took a look at the performance (from https://github.com/IntelRealSense/librealsense/pull/1866):
Initial CUDA support for conversions and pointcloud generation.
Performance improvement:
As tested on the Jetson TX2:
Description Resolution fps No Cuda, avg. (ms) With Cuda, avg. (ms) YUY2 into RGB8 1920×1080 30 30 – 32 5-7 YUY2 into RGBA8 1920×1080 30 34 7.5 YUY2 into BGR8 1920×1080 30 35 5 – 7 YUY2 into BGRA8 1920×1080 30 37 7 – 10 YUY2 into Y16 1920×1080 30 10 5 – 8 YSplitting Y8I to Y8, Y8 1280×800 30 9 – 10 3 – 5 Splitting Y12 to Y16, Y16 1280×800 25 22.9 5 – 7 Pointcloud (deproject depth) 640×480 30 8 – 9 3 – 5 In order to activate GPU full power run: sudo ./jetson_clocks.sh
You can see that there is a 4-7x speedup on the CPU only versus GPU code on the Jetson. The term fps means frames per second. Remember that when you are running 30 fps, each frame arrives about every 33ms.
Note: Because space is tight on the Jetson TX1, you should first flash the Jetson TX1 with L4T 28.2 using JetPack 3.2.1, download the buildLibrealsense2TX repository, and then build the patched kernel. Once the kernel is in place, remove the kernel sources to give you enough space to operate.
Installation
On the JetsonHacks Github account, there is a repository buildLibrealsense2TX. To download the repository:
$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
$ git checkout v0.9
The instructions are much the same as in the previous article. More details are available on the README.md file in the buildLibrealsense2TX directory.
Explain please
First, a word about what we’re doing. There are several RealSense camera formats that the standard UVC (video) module in the kernel does not support. librealsense provides patches to add those video modes to the appropriately. The patches also add support for properly adding timestamps to the incoming video stream. In addition, there are several patches that modify some of the Industrial I/O (IIO) tree. These patches are providing support to supplemental hardware on a RealSense camera, such as a 3D Gyro or 3D Accelerometer.
Note: Some of the patches apply to modules that are built into the kernel itself. Because these modules are required to be in the Image, and not built as external, you will need to compile the kernel itself along with the modules.
If you do not modify the Linux kernel, librealsense will mostly appear to work. You will probably experience issues a little further along when you are trying to get better precision and accuracy from the video and depth streams. As the RealSense D435 camera does not have supplemental hardware, the IIO patches don’t make any difference.
If you’re just looking to play around with the camera, you may be able to get away with not compiling the kernel and skip over it. If you’re more serious, you’ll have to start patchin’ !
Building the Kernel
Note: If you built your kernel as in the previous article, you must rebuild it again!
Building the kernel can be taxing, there are many little things that can go wrong. Plus, you can make your Jetson become brickly if something happens at an untoward momement. If you’re a go for it kind of person:
$ ./buildPatchedKernelTX.sh
If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.
$ ./buildPatchedKernelTX.sh –nocleanup
Actually, this script is more useful as a template for rebuilding your own kernel with the librealsense changes. There are two scripts in the ‘scripts’ directory for helping in the kernel build:
patchKernel.sh applies the librealsense kernel patches.
configureKernel.sh Configures the kernel to add the appropriate modules needed by librealsense.
Since you’re a developer, you should be able to figure out what the scripts are doing and modify them to match your needs.
librealsense Installation
make sure that the RealSense camera is not attached to the system. Then install the library:
$ ./installLibrealsense.sh
The script does several things:
- Installs dependencies
- Builds CMake
- Builds librealsense and associated apps
CMake 3.8+
The librealsense build system requires CMake version 3.8 or above. The standard Jetson TX version of CMake is 3.5.1 for L4T 28.2.x. Therefore the script builds CMake (in the home folder). However, the script does not install CMake into the system area, it just uses the new version directly from the build fold. Of course, you can install the new version of CMake on your system if so desired.
After the script compiles the library, the new files are placed in the following directories:
- The library is installed in /usr/local/lib
- The header files are in /usr/local/include
- The examples and tools are located in /usr/local/bin
The script also sets up a udev rule so that the RealSense camera is available in user space.
Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:
$ cd /usr/local/bin
$ ./realsense-viewer
Conclusion
It is nice to have the CUDA upgrade for librealsense in the official repository. Because the RealSense SDK is under heavy development, we will have to keep our eye out for improvements in the weeks ahead!
Notes
- In the video, a Jetson TX2 is shown
- L4T 28.2.1 installed by JetPack 3.2.1
- buildLibrealsense2TX is version v0.9
Intel Documentation:
28 Responses
Will this work with the R200 3D camera as well as the D400’s? I ordered a 435 back in January and so far have been unable to obtain one.
This update is for librealsense 2, which is only for the new cameras. Thanks for reading!
Does the kernel rebuild really need to be done on a freshly flashed system only?
It depends. When it is freshly flashed, the system is in a known state and the procedure described in the articles works. If you are comfortable building your own kernel, then you probably already know where any possible build issues may lie. For simplicity in these articles, we assume a fresh flash just to eliminate trying to debug different configurations and dependencies. Thanks for reading!
Hi,
I am new to Linux and Jetson platform. I am planning to use TX2 on Auvidea J120 carrier board and attach RealSense D435 to it. Now for enabling USB 3.0 on the carrier board, the vendor has provided some dtb files to flash. From the article above I understand that I need to rebuild kernel as well.
I went through the Kernel build scripts you provided and notice that it is slightly different from the official docs. In official docs they build kernel, dtb and modules and then copy the kernel image and dtbs to respective folders in Linux_for_Tegra/kernel and Linux_for_Tegra/kernel/dtb (for later flashing via JetPack I suppose). At the end they use make modules_install to install modules in Linux_for_Tegra/rootfs.
In your scripts you are building on the platform but then you just replace the Image in /boot. What about the dtbs? shouldn’t these be copied somewhere? and I noticed in makekernel.sh at the end you also execute make modules_install. Does it copy the modules to rootfs as well?
I am planning to cross-compile on host to build the kernel for librealsense. So I am trying to go through your scripts to understand and perform the kernel config, patch and build manually step by step. If you can explain the differences I mentioned above between your post build steps and the official docs it will immensely help. Sorry for the long post.
Thanks
Second question, since I will be replacing some dtbs from the carrier board (Auvudea J120) vendor in the Linux_for_tegra/kernel/dtb folder, now if I build the kernel as well (for realsense) how can I merge the dtbs from the kernel build and ones provided by the carrier board vendor? I am stuck. Sorry if it sounds stupid but like I said this is my very first project in Linux and Jetson so most of the time I am not sure what to do 🙂
You should probably ask for guidance from Auvidea or ask this question on the official NVIDIA Jetson forum where a large number of developers and NVIDIA engineers share their experience. The forum is here: https://devtalk.nvidia.com/default/board/188/jetson-tx2/
To be clear, this is not a good project for an introduction to Linux or the Jetson 😉 You can probably use the D435 without recompiling the kernel until you hit some of the bottlenecks outlined in the article.
I have tried Auvidea support, its not good. They don’t respond. NVIDIA forums are great but the opinions are so mixed up that its taking me time to decipher everything, but I am doing it.
I agree its not a good starting point but I don’t really have an option. It is my job requirement so have to deal with it 🙂
Yea I think I will go ahead without recompiling the kernel for now and first get everything running, the carrier board and D435.
One last question, If I don’t rebuild the kernel, the librealsense would still work with CUDA enhancements? I know it has got nothing to do with the kernel but better to ensure.
Thanks
Your assessment is correct. The kernel modifications have to do with making a wider variety of video modes available in user space, and support for hardware which is not present on the D435.
Yea Intel is planning a new variant D435i with an IMU embedded. Launching towards the end of year. So may be these modules will be useful in the future.
Hi Again,
I thought to give it a try and build the kernel for learning purposes. I am cross-compiling it on a non-tegra machine. I am following your configure and patch scripts. Here are the steps
1- Downloaded the kernel source and librealsense source
2- extracted current .config from TX2, moved it to host and configured it using your configure script
3- Patched kernel using the patchkernel script
4- ran make module_prepare, make module, make module_install. It all went fine.
5- ran make Image and it failed with following message
LD init/built-in.o
drivers/built-in.o: In function `nvs_remove’:
/home/nouman/tegra-build/kernel/kernel-4.4/drivers/misc/nvs/nvs_iio.c:1804: undefined reference to `devm_iio_kfifo_free’
drivers/built-in.o: In function `nvs_init’:
/home/nouman/tegra-build/kernel/kernel-4.4/drivers/misc/nvs/nvs_iio.c:1846: undefined reference to `devm_iio_kfifo_allocate’
/home/nouman/tegra-build/kernel/kernel-4.4/Makefile:959: recipe for target ‘vmlinux’ failed
make[1]: *** [vmlinux] Error 1
make[1]: Leaving directory ‘/home/nouman/tegra/kernel’
Makefile:150: recipe for target ‘sub-make’ failed
make: *** [sub-make] Error 2
Now the errors above look like they are related to the modules enables in the .config file earlier. Can you spot any obvious issues with my process? I remember while updating the .config file, there were modules related to IIO KFIFO BUFFER and the errors are to do with undefined references to iio_kfifo_buffer allocate and free. I looked through the source files and I couldn’t find these references defined anywhere. However, I did find them declared in one of the .h files included in main .c file where the error occured.
Any pointers you can throw at me would be highly appreciated.
Thanks
Nouman.
I don’t know too much about the cross compile process. It looks like you might be missing a module. Please ask this question on the official NVIDIA Jetson forum where a large number of developers and NVIDIA engineers share their experience. The forum is here: https://devtalk.nvidia.com/default/board/188/jetson-tx2/
cheers Kangalow!
You may need to change CONFIG_IIO_KFIFO_BUF from ‘m’ to ‘y’
Not quite sure I understand the question. The compiled linux kernel is the file “Image”. The Linux_for_Tegra/rootfs is a direct disk image of what will be placed on the Jetson during flashing. Compiling the kernel from source will build a new Image file which replaces the Image file on the disk image (ie Linux_for_Tegra/rootfs/boot/Image). On a running system, this is simply the file /boot/Image.
modules_install is a script that is part of the kernel build process which basically copies the module files into /lib/modules/(name of the kernel)/kernel You can find these on a host in Linux_for_Tegra/rootfs/lib/modules/(name of the kernel) where name of the kernel is something like ‘4.4.38-tegra’. On a running system, this is simply /lib/modules/(name of the kernel).
To be clear, JetPack is the name of the program/installer that flashes the disk image to the Jetson. L4T is the name of the operating system, which is an Ubuntu variant.
The dtbs in the more recent Jetson releases is not stored on the rootfs partition. Instead, it is encoded and placed in a separate device tree partition on the disk. Currently the encoding is done using an x86 binary, so it cannot be done on the Jetson itself. During the flashing process, JetPack copies the compiled device tree files to the device tree partition. There is a command line script utility which you can use on the host to flash only the device tree partition, I believe that it is called “flash” and requires some command line parameters and flags to work correctly.
Hope this helps.
Many thanks for the detailed response. I think I understand better now. Obviously not fully comprehend everything but I am starting to build a mental model that kinda makes sense 🙂 You are a star 🙂
Thank you for the kind words.
Thank you for the amazing post.
I’m wondering if SDK support body-pose estimation, and face contour estimation?
like the one here: https://www.youtube.com/watch?v=Ft2mMnAD7N4
Thanks again for the helpful content.
Thank you for the kind words. As far as I know, librealsense is only a driver for the camera, and does not perform the application specific functions that you mention. Thanks for reading!
Does it work when i dont have CUDA installed on my JETSON TX2? I get the following error:
— Info: REALSENSE_VERSION_STRING=2.10.4
CMake Error at /home/nvidia/CMake/Modules/FindCUDA.cmake:687 (message):
Specify CUDA_TOOLKIT_ROOT_DIR
Call Stack (most recent call first):
CMakeLists.txt:61 (find_package)
— Configuring incomplete, errors occurred!
See also “/home/nvidia/librealsense/build/CMakeFiles/CMakeOutput.log”.
Building librealsense, headers, tools and demos
make: *** No targets specified and no makefile found. Stop.
I dont have it installed, because somehow the installation through Jetpack 3.2.1 wont work.
I would be glad, if you could help me out. I am really stuck here.
Thanks in advance.
Please examine the script files and modify them to suit your application. There is a flag you can turn off which does not include CUDA. However, you should fix you CUDA installation.
Is this still necessary in librealsense 2.16.0+? One of the Bug fixes was kernel 4.15 support.
Not sure I understand the question. On the Jetson TX2, the kernel is 4.4
Thanks for reading!
Hi,
I followed all the explanation here (great scripts, I built the kernel and the librealsense) and now the RealSense depth cameras (D415 or D435) are working on the Jetson TX2 but only once at a time.
If I connect both as I need it (with a powered usb 3.0 hub) even the realsense-viewer fails after a few seconds. The depth map freeze and appear the error: “Hardware Notification #N Frames didn’t arrived within 5 seconds”.
Have you tried with 2 RealSense sensors at the same time?
I have L4T 28.2.1 and Jetpack 3.3
Thanks in advance
I am glad you found the scripts useful.
I have not tried two cameras at once on the Jetson TX2. However, I do not believe that there is enough bandwidth on the Jetsons one USB bus to run both at the same time. You may want to check on the Jetson TX2 forums to see if anyone else has gotten it to work. Thanks for reading!
I am doing a fresh installation using the Realsense D435 with Jetson TX2 and L4T 32.2/Ubuntu 18.04. The scripts provided in this tutorial are not working with this new version but after studying the source code I think they can be easily changed to use the new version of the kernel, the scripts are really well-structured. Are you thinking in creating a new version for the scripts in order to support this new L4T version for Jetson TX2?
Thanks
I can’t say that it’s as formal as a plan, though I intend to spend some time in the fall updating the older scripts on the Jetson TXs and Xavier. Thanks for reading!
I have given a try by myself but It is more difficult than expected. The kernel patch provided by the librealsense for the bionic distribution assume a kernel 4.15, but the kernel comming with the L4T 32.2 is 4.9. For this reason the parches fails. I have also checked the differences and they are not trivial at all. I am stuck at this point. I have reviewed the kernel supported version in different version of the librealsense and it looks they are not supporting 4.9. Since I am a little bit in a hurry I am going to switch the instalation to L4T 28.2, hoping that the librealsense will be available for the L4T 32.2 in the near future.
Thanks