Intel has recently begun shipping the RealSense D435 and D415 depth cameras. Let’s start working on running them on the NVIDIA Jetson TX kits. Looky here:
As you may recall, we were using librealsense with the previous generation R400 RealSense camers. With the advent of the new D400 series RealSense cameras, Intel has upgraded the librealsense to version 2.0 to support the new camera family and its features.
The new hardware introduces a couple of different video modes, as well as support for camera on-board accelerometer and gyroscope. While the D435 in the video does not have the additional hardware, other cameras in the range do. As a result, librealsense requires modification to the Jetson kernel Image and additional modules to support the new features.
The Jetson TX kits are embedded systems, so they don’t quite line up with the way that most developers think about the Linux desktop. In the regular installers for librealsense, there are several assumptions made about how devices attach to the system. Also, some assumptions are made about the kernel configuration that do not match the Jetson.
The bottom line is that we need to build a new kernel to support the RealSense cameras. We’ll break this installation into two parts. The first part is installing librealsense itself. The second part will build a kernel that supports the cameras.
On the JetsonHacks Github account, there is a repository named buildLibrealsense2TX. To download the repository:
$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
Next, make sure that the RealSense camera is not attached to the system. Then install the library:
Looking inside the script file, you will see that there are a couple of patches for the Jetson. The first patch is a work around for some code related to an Intel specific instruction set, the other is a workaround for the Industrial I/O (IIO) device detection.
The stock librealsense code appears to assume that a IIO device reports with a device and bus number. On the Jetson, the ina3221x Power Monitors do not follow this protocol. The result is that a series of warning are issued continuously as the library scans for HID devices that have been added (plugged in) to the system.
The library is looking for IIO HID devices (the accelerometer and gyroscope on a RealSense camera). The ina3221x is not a HID device, but appears during the IIO scanning. The library scans, but because it does not find a device or bus number for the power monitor, it issues a warning to stderr (the console). The result is that the console gets spammed, which in turn results in a performance penalty.
The workaround patch checks to see if the device detected is the power monitor before issuing a warning.
Similarly, the built in camera module control is via CSI/I2C, not USB as expected by librealsense. Again, a warning is sent to stderr by librealsense. There may be a clever way to determine if the warning is about the on-board Jetson camera, but in this case the patch just comments out the warning.
After applying the patches, the script compiles the library, examples and tools:
- The library is installed in /usr/local/lib
- The header files are in /usr/local/include
- The examples and tools are located in /usr/local/bin
The script also sets up a udev rule so that the RealSense camera is available in user space.
Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:
$ cd /usr/local/bin
As shown in the video, you will be able to use the camera. However, if you examine the console from where the app is launched, you will notice that there are a couple of issues. First, some of the video modes are not recognized. Second, some of the frame meta-data is absent.
The video modes are identified in the Linux kernel. The frame meta-data is too. In order for this information to become available to librealsense, patches must be applied to the kernel and the kernel rebuilt.
You will need to make the determination as to if the added information is important for your application.
Kernel and Modules
The changes that librealsense needs are spread across several files. Some of the changes relate to the video formats and frame meta-data. These changes are in the UVC video and V4L2 modules.
In previous versions of librealsense, we would build the UVC module as an external module. This was relatively simple. However, things have changed a little internally in the way that L4T 28.2 is configured. The V4L2 module is built into the kernel Image file (it is an ‘internal’ module). The UVC can still be compiled as an external module.
The other new HID modules that the library uses are part of the IIO device tree. These modules rely on the internal module for IIO, as well as a couple of other support modules which must be enabled as internal modules.
As a result, this is a little tricky for development purposes in a general purpose manner. First, there are some modules which need to be enabled. They are a little picky in that some need to be internal modules. There are also patches that need to be applied to the stock kernel sources.
There are two ways to go about this. The first is the recommended way, where you work it all into your development process. The second way is to use a provided script which will attempt to build a stock kernel with the addition of the kernel support needed for librealsense. If something goes wrong during this second method, most likely you will be forced to reflash your Jetson because it is in a bad state.
In either case, you should be building your kernel on a freshly flashed Jetson. You will need ~3GB of free space for building the kernel.
We’ve talked about building the kernel with modules before. Basically the steps are:
- Get the kernel sources
- Configure the kernel
- Apply any needed patches
- Make the kernel and modules
- Cross your fingers and hope
Typically you’re working on a development kit, where you have your own kernel modification and configuration in place. You will need to do some configuration of the kernel for librealsense. Located in:
buildLibrealsense2TX/config/ (e.g. buildLibrealsense2TX/config/TX2
there is a stock .config file with the changes that are needed to the configuration for the librealsense library. You should diff this with a stock kernel configuration, and then add the changes to your development .config file.
You can then apply the kernel patches:
These patches will fix up the camera formats, add the meta-data and so on.
At this point, you are ready to build the kernel and install it. As usual, you should make a backup of the stock image and modify /boot/extlinux/extlinux.conf to add an option to boot to either the stock image or the new image. Also, remember to set the local version in the .config file!
Just go for it!
You’re the type of person who lives on the edge. Doesn’t care about what others think. You just want it, and you want it now. Have I got the script for you!
If you are not concerned about kernel development and just want the camera up and running properly, you can run a script which will rebuild the kernel with all of the changes needed and install it. Just to be clear, this will install a stock kernel with the librealsense changes in place of whatever is currently there. If you have kernel modifications already installed, they will disappear.
Be forewarned though, sometimes when you live on the edge, you can fall over the edge. If something untoward happens during the build, it can render your Jetson brickly; you will need to reflash it.
For the install on a Jetson TX2:
After the installation, reboot, and you should be ready for goodness.
Note: We’ll provide a TX1 script soon.
If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.
$ ./buildPatchedKernelTX2.sh –nocleanup
Actually, this script is more useful as a template for rebuilding your kernel with the librealsense changes.
As we note in the video, there appears to be some issues with performance with the realsense-viewer application. It seems rather suspicious that when the program starts that 100% of one of the CPU cores is being used. This usually indicates that the GUI is not yielding at the bottom of its event loop.
Another issue is that most of librealsense is built for optimization of x86 code. This is to be expected, after all it is an Intel product. Because the Jetson is ARM based, the code defaults to some generic drivel. If only there was a way to exploit parallel processing on a Jetson which has 256 CUDA cores …
Making progress! Off to the next installment of the series.
- In the video, iInstallation was performed on a Jetson TX2 running L4T 28.2 (JetPack 3.2)
- Librealsense 2.10.2
- Intel RealSense D435 camera