Intel is now shipping the RealSense D435i Depth Camera. The camera includes a built in IMU. In this article, we cover interfacing with the Jetson AGX Xavier. Looky here:
Background
In previous articles, we went through how to install the Intel RealSense library (called librealsense 2) on the Jetson TX1 and Jetson TX2. Since that time, we have seen the introduction of the RealSense D435i camera and Jetson AGX Xavier.
The software drivers to interface with the D435i have seen a few updates (such as better CUDA support) since that time. However, the biggest challenge in installing full librealsense 2 support on the Xavier is that additional kernel modules must be built and installed. Because some of the affected modules are built in to the kernel itself, the kernel image itself should be built.
The Jetson AGX Xavier is an embedded system. In this implementation, the Linux kernel is signed and resides in a partition on disk. This is for security reasons, as you are aware if you have been following the computer news for the last few months. This complicates building the kernel on the Jetson Xavier itself, as the signing application only runs on a PC. The JetPack installer contains the signing application.
The NVIDIA approved method is to cross compile the kernel and modules on the PC, and then flash them on the Jetson. In this article, we do something exactly unlike that.
Librealsense 2 Installation
Note: This installation is for intermediate and advanced developers. You should be familiar with kernel modules, and the kernel Image. It is strongly suggested that you do this on a fresh install, immediately after flashing the Jetson. You should make backups of anything value, by you know that already.
WARNING: This installation method assumes that you are running a stock kernel.
On the JetsonHacks Github account, there is a repository named buildLibrealsense2Xavier. To download the repository:
$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2Xavier
$ cd buildLibrealsense2Xavier
Build Kernel and Modules
The first step is to build the needed modules and a new kernel. Also, in order to have the Xavier understand the different video formats, there are some patches to apply to the module source code.
There are no librealsense 2 patches available to directly patch the Xavier, because it is running kernel version 4.9. This is between the different kernel versions which have patch revisions. For this reason, the patches sub-directory contains artisan patches, exquisitely crafted to upgrade the Jetson Xavier to run the librealsense 2 libraries.
Some of the patches actually change header files which touch some modules which are built in to the kernel. That is why the kernel Image needs to be updated. If the kernel Image is not updated, the logs get cluttered with a large number of warnings about incorrect video formats being present.
In order to patch and rebuild the kernel Image, modules and then install the modules:
$ ./buildPatchedKernel.sh
At the end of the process, you should have an ‘Image’ file in the image sub-directory.
Note: There is a convenience script removeAllKernelSources.sh which will remove all of the kernel sources that the buildPatchedKernel script downloaded, along with all of the build remnants. If you do not intend to build other kernel modules, or modify the kernel Image itself, you may find this a useful tool.
Flashing the Kernel Image
Once the script is finished, you can now flash the new Image on to the Jetson. Use a file transfer utility such as scp or ftp to transfer the new Image file to the host PC with the JetPack installer. Sneakernet works too.
Now shutdown the Xavier. Connect the Jetson to the host PC via the USB cable in the same manner as the original JetPack flash. Then place the Xavier into Force Recovery Mode. This is also the same procedure used when flashing JetPack.
Go to the host PC. Make sure you make a backup of the original Xavier kernel Image. The Image should be in the host JetPack directory, in the Xavier/Linux_for_Tegra/kernel directory. With the backup secure, you can then copy the new Image that was built on the Jetson in its place.
Then go up one level to the Linux_for_Tegra directory. You are then ready to flash the Image:
$ sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1
This will flash only the kernel Image to the Xavier, replacing the previous version. This should not effect the rootfs. The Xavier will reboot once the kernel Image has been transferred. Go back to the Xavier.
Build librealsense 2
The machine has just rebooted. Open a terminal and go back to the repository directory.
$ cd buildLibrealsense2Xavier
Make sure there are no RealSense cameras attached to the Xavier. Now build librealsense 2, and install the libraries and applications:
$ ./installLibrealsense.sh
The script compiles the library, examples and tools:
- The library is installed in /usr/local/lib
- The header files are in /usr/local/include
- The examples and tools are located in /usr/local/bin
The script also sets up a udev rule so that the RealSense camera is available in user space.
Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then execute the tools and examples, such as:
$ cd /usr/local/bin
$ ./realsense-viewer
Conclusion
This installer is still a work in progress. There are a couple of issues that need some work. However, enough people have asked about this that it seems worthwhile to get something out there for people to start working with.
Notes
- In the video, installation was performed on a Jetson AGX Xavier running L4T 31.1 (JetPack 4.1.1 Developers Preview).
- librealsense v2.17.1
- buildLibrealsense2Xavier v0.8
51 Responses
Are you able to get the realsense camera to work when connected directly to the Xavier’s USB ports? I got errors like “Incomplete video frame detected” and “USB SCP overflow” when I use the dev kit’s built in USB ports. If I use a PCIe USB card it works fine. Trying to figure out if I got a bad unit or if they are all like this.
Thanks for the great post, the kernel build script works perfectly.
I have the same issue on this Xavier as you encountered. The whole kernel build process is a little convoluted here, but it feels simpler to me than setting up the cross compile environment and running it on the PC. Which PCIe USB card are you using? Did you have to compile a kernel module to recognize it?
I’m guessing that this has to do with the USB firmware, I’m a little surprised that the USB-C ports act differently than the USB 3.1. I tried a USB-C hub, but that did not appear to help matters; it acts the same as when the camera is plugged directly in to the USB-C port.
I’m guessing that it has something to do with the RealSense Camera acting like a block-transfer device. There’s probably some timing or handshake issue down at the firmware level. Fortunately I don’t have access to that code or the tools needed, because that’s real work to debug!
Thanks for reading!
Have the same issue. Thought that it’s been caused by this kernel’s patch, but after rolling back to original kernel image, found the same errors, whether connecting to Type-C or Type-A USB port, or USB hub.
Though I hadn’t had so many errors before. So, probably, the reason is the latest firmware (05.11) and/or librealsense version (2.19).
I used the HighPoint Technologies RocketU 1322A. (marked RU1322A on pcb) It cost me around 50 USD and it worked without any special kernel module. The card does stick out the top rather precariously, ok for desktop use but not mechanically robust enough for a robot.
Please.. JETSON TX2 + Jetpack 3.3 + D435i install guide..
I followed your install guide for installing a realsense with Jetson tx2 with Jetpack 3.2 but I failed.. I reset my jetson tx2 with Jetpack 3.2 before following your install guide. And I tried with Jetpack 3.3 as well. But the camera always not detected when I activate the realsense-viewer. The camera was connected with usb 3.0 port.
Do you think it is happened due to my camera is D435i, not D435?
If you use Jetpack3.3, the version of Librealsense should be 2.13 which may probably not support D435i. Try D435, that would be OK. If you wanna use D435i, I’m afraid you have to update your Jetpack.
Hi. Has anyone attempted to use more than one D435 cameras (synced) for wider field of view with Xavier? How does Xavier compare with other embedded computers for robotic applications (with ROS)?
Hello,
Has anyone tried this camera with digits for object detection?? https://forums.intel.com/s/question/0D50P00004FzJ2zSAF/is-intel-realsense-d435i-vl42-compliant?language=en_US
How long does it take to compile the kernel on Xavier (after running ./jetsonClocks.sh)
I noticed that the default kernel of Jetpack 4.1.1 has PREEMPT enabled, does your custom kernel also have this enabled?
nvidia@jetson-0423018054325:~/dev/rt-tests-1.3$ uname -a
Linux jetson-0423018054325 4.9.108-tegra #1 SMP PREEMPT Wed Oct 31 15:17:21 PDT 2018 aarch64 aarch64 aarch64 GNU/Linux
It takes ~ 19 minutes.
The kernel build uses the current running configuration:
$ zcat /proc/config.gz > .config
The .config file sets all of the kernel and module information. I haven’t looked specifically at PREEMPT, but would be very surprised if it was different.
Small issue in the description above: it says twice
cd buildLibrealsense2Xavier
cd buildLibrealsense2Xavier
But you want
cd buildLibrealsense2Xavier
./installLibrealsense.sh
Good catch! I appreciate the help.
Is there a way to run Xavier on Ubuntu 16.04?
Not that I know of. Please ask this question on the official NVIDIA Jetson forum where a large number of developers and NVIDIA engineers share their experience. Jetson AGX Xavier forum: https://devtalk.nvidia.com/default/board/326/jetson-agx-xavier/
Good job, thanks!
One thing to note is that librealsense/CMake/cuda_config.cmake seems to be better modified with Xavier’s arch ‘compute_72’ instead of ‘compute_53’ and ‘compute_62’ currently used, which probably stand for TX1/2.
Is there any plans to make a similar process available for the new Jetpack for TX2 and Nano?
dude,can you install D435i driver on TX2(Jetpack4.2)successfully?
Yes nano support would be greatly appreciated!
Is the nano gonna be supported any time soon?
Thanks for the video and also the review of the TX combo! Do you think this would work on a nano for static scenes mapping appilcations?
After connecting directly to the Xavier’s USB ports. I met the Depth stream freezes with error “USB SCP overflow” problem. But it seems OK in the tutorials.Is there any solution?
In the videos, I went through a USB 3.0 Hub.
Thank you for reply.
One more detail, where is the USB 3.0 Hub connected? Xavier usb port, type-c to usb port or pcie to usb port?
On the USB 3.0 dual purpose connector.
It works on the 3.0 hub!
It is still weird that 3.0 hub works but directly connect to the usb3.0 port on Xavier is not. They are both usb3.0 port.
Yes, I have found the Xavier is a little finicky in this regard. It could be that the connector doesn’t have enough power.
Try disabling LPM on D430/D435/D435i in usb core.
Although you will still get “Incomplete video frame detected” and “USB SCP overflow” errors for the usecases which require lots of bandwidth, the depth camera should work fine then.
Check https://devtalk.nvidia.com/default/topic/1048804/jetson-agx-xavier/realsense-camera-unstable/post/5364499/#5364499, this should be included in the next Jetpack release.
Who can install D435i driver on TX2(Jetpack4.2+kernel4.9+librealsensev2.22) successfully?
In order to use this guide on TX2(Jetpack4.2+kernel4.9.140+librealsense_v2.22),I have changed some code.Everything is ok but applying realsense metadata patch.There are some error I get.
patching file drivers/media/usb/uvc/uvcvideo.h
Hunk #2 FAILED at 222.
1 out of 2 hunks FAILED — saving rejects to file drivers/media/usb/uvc/uvcvideo.h.rej
How can I fix it?I never change anything under patches folder.
You would need to apply the patches for the 4.9 kernel. An example is in the pull request here: https://github.com/jetsonhacks/buildLibrealsense2Xavier/pulls
Hey Kangalow, I followed the example for the 4.9 Kernel but I think I missed an option somewhere along the lines.
I would like to use the Python wrapper (Pyrealsense2) but import doesn’t work.
I am using python 3.7 , Any help would be appreciated !
How did you set your PYTHONPATH environment variable to include librealsense?
I think I am misunderstanding this question. If you mean this : export PYTHONPATH=$PYTHONPATH:/usr/local/lib
I did that, and seems to have no effect.
Did you try looking to see where they actual pyrealsense .so file is?
Yeah I think, pyrealsense2.cpython-37m-aarch64-linux-gnu.so
pyrealsense2.cpython-37m-aarch64-linux-gnu.so.2
pyrealsense2.cpython-37m-aarch64-linux-gnu.so.2.xx.0
Are in usr/local/lib and in /home/MYNAME/librealsense/build/wrappers/python
My bashrc file has export PYTHONPATH=$PYTHONPATH:/usr/local/lib
as last line.
When I run echo $PYTHONPATH
:/home/MYNAME/librealsense/wrappers/python/pyrealsense2:/home/MYNAME/librealsense/build/wrappers/python:/usr/local/lib
It just breaks my brain at this point, hah.
Still stuck on this, I’ve tried editing the installlibrealsense.sh with these lines :
cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true -DPYTHON_EXECUTABLE=/usr/bin/python3.7 -DBUILD_EXAMPLES=true -DBUILD_WITH_CUDA=true
But that did not help. I hope you can hand some information on how to get this python wrapper working. (I saw you did a video on ROS-wrapper already)
I used TX2+Jetpacke4.2+D435i,it worked but the issue you mentioned existed. I tried to turn to a low resolution, it works well. So I am afraid the TX2 could not deal with high resolution frame. Actually I am waiting for someone to help me with the high resolution frame.
Hello kangalow , I have watching all of your video from the frist time you introduce the intel realsense camera to this article . And there is a question , I have a TX2 and a intel D435i camera . Your the last video about how to install realsense camera is published at 20180710 with L4T 28.2.1/JetPack 3.2.1/librealsense v2.13.0. And D435i was launched at oct 2018 and support by intel begin with 20181025/librealsense v2.16.2. In this article it is v2.17.1.
And there is a question , can I just follow this article to install D435i driver on TX2 ? If not ,what and how should I do ?
And another question is the newer L4T always base on ubuntu 18.04? Can I using the newest JetPack to flash ubuntu 16.04 to my TX2 ?
Does that lib supports and the realsense T265 on xavier or not?
Please refer to the librealsense Github repository for camera support: https://github.com/IntelRealSense/librealsense
So it turns out I did everything right.
Except for one thing! I needed to reboot after the EXPORT step.
Now import works. Thanks for all the help again.
Let me know what items you are looking at for your next project.
I can have my company ship a few products to give back for all you do for the community. Send me an email 🙂
I am glad you were able to get it to work. Seems a little tricky to get working.
I got the version mismatch error when build the kernel
VIDIA Jetson AGX Xavier
Jetpack UNKNOWN [L4T 32.3.1]
Jetson AGX Xavier Development Kit
==== L4T Kernel Version Mismatch! =============
This repository is for modifying the kernel for a L4T 32.1.0 system.
You are attempting to modify a L4T 32.3.1 system.
The L4T releases must match!
There may be versions in the tag/release sections that meet your needs
My jetpack version is 4.3
For version JetPack 4.3, Intel recommends installing from their APT repository. See: https://github.com/jetsonhacks/installRealSenseSDK
For version JetPack 4.3. I tried https://github.com/jetsonhacks/installRealSenseSDK . It works for me .
The problem is when I use multiple cameras, bugs occurs.
As described in https://github.com/IntelRealSense/librealsense/issues/5828.
I have a D435i and a D415 camera with both firmware 05.12.03.00.
First , the cameras both connected to the same usb3.0 hub on a ubuntu16.04 PC, it is OK to run `rs-multicam`.
Second, the cameras both connected to the same usb3.0 hub on Xavier with Jectpack 4.3 .
If using the realsense-viewer, I can see both camera stream , but if using c++ api `rs-multicam` ， it crashed.
The test code is
“`
rs2::context ctx; // Create librealsense context for managing devices
std::map colorizers; // Declare map from device serial number to colorizer (utility class to convert depth data RGB colorspace)
std::vector pipelines;
std::cout<<"test0 ==== "<<std::endl;
// Start a streaming pipe per each connected device
for (auto&& dev : ctx.query_devices())
{
rs2::pipeline pipe(ctx);
rs2::config cfg;
cfg.enable_device(dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER));
pipe.start(cfg);
std::cout<<"test1 ==== "<<std::endl;
pipelines.emplace_back(pipe);
// Map from each device's serial number to a different colorizer
colorizers[dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER)] = rs2::colorizer();
}
“`
The result is
“`
test0 ====
test1 ====
terminate called after throwing an instance of 'rs2::error'
what(): failed to set power state
Aborted
“`
disable this line
`cfg.enable_device(dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER));`
fix my problem.
Excellent, I am glad you got it to work.
I have a Jetson AGX and a D435i which I am trying to work with ISAAC SDK 2021.1.
ISAAC SDK application can’t show the camera feed so I was trying to patch the Kernel and rebuild the image. I have Jetpac 4.5 and
Linux expertbot-desktop 4.9.201-tegra #1 SMP PREEMPT Wed May 5 09:32:25 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux
I got the following error when rebuilding it through ./buildPatchedKernel.sh
If anyone has a similar issues and fixed it then please help.
/usr/src/kernel/nvidia/drivers/video/tegra/dc/hdcp/hdmihdcp.c: In function ‘link_integrity_check’:
/usr/src/kernel/nvidia/drivers/video/tegra/dc/hdcp/hdmihdcp.c:1940:16: error: unused variable ‘nonce’ [-Werror=unused-variable]
unsigned char nonce[HDCP_NONCE_SIZE];
^~~~~
CC drivers/usb/gadget/function/uvc_queue.o
LD drivers/video/tegra/camera/built-in.o
CC drivers/video/tegra/dc/hdcp/nvhdcp_hdcp22_methods.o
CC drivers/usb/gadget/function/uvc_v4l2.o
CC drivers/video/tegra/host/nvcsi/nvcsi.o
cc1: all warnings being treated as errors
scripts/Makefile.build:335: recipe for target ‘drivers/video/tegra/dc/hdcp/hdmihdcp.o’ failed
make[4]: *** [drivers/video/tegra/dc/hdcp/hdmihdcp.o] Error 1
scripts/Makefile.build:649: recipe for target ‘drivers/video/tegra/dc’ failed
make[3]: *** [drivers/video/tegra/dc] Error 2
make[3]: *** Waiting for unfinished jobs….
CC drivers/video/tegra/host/nvcsi/deskew.o
CC drivers/usb/gadget/function/uvc_video.o
LD drivers/usb/gadget/function/usb_f_uvc.o
LD drivers/usb/gadget/function/built-in.o
CC drivers/video/tegra/host/nvcsi/nvcsi-t194.o
LD drivers/usb/gadget/built-in.o
LD drivers/video/tegra/host/nvcsi/nvhost-nvcsi.o
LD drivers/usb/built-in.o
CC drivers/video/tegra/host/t124/t124.o
CC drivers/video/tegra/host/t186/t186.o
LD drivers/video/tegra/host/nvcsi/nvhost-nvcsi-t194.o
LD drivers/video/tegra/host/nvcsi/built-in.o
CC drivers/video/tegra/host/t194/t194.o
LD drivers/video/tegra/host/t124/nvhost-t124.o
LD drivers/video/tegra/host/t124/built-in.o
CC drivers/video/tegra/host/t210/t210.o
LD drivers/video/tegra/host/t186/nvhost-t186.o
LD drivers/video/tegra/host/t186/built-in.o
CC drivers/video/tegra/host/vi/host_vi_notify.o
LD drivers/video/tegra/host/t194/nvhost-t194.o
LD drivers/video/tegra/host/t194/built-in.o
CC drivers/video/tegra/host/vi/vi4.o
CC drivers/video/tegra/host/vi/vi_notify.o
CC drivers/video/tegra/host/vi/vi5.o
LD drivers/video/tegra/host/t210/nvhost-t210.o
LD drivers/video/tegra/host/t210/built-in.o
LD drivers/video/tegra/host/vi/nvhost-vi5.o
LD drivers/video/tegra/host/vi/built-in.o
LD drivers/video/tegra/host/built-in.o
scripts/Makefile.build:649: recipe for target ‘drivers/video/tegra’ failed
make[2]: *** [drivers/video/tegra] Error 2
scripts/Makefile.build:649: recipe for target ‘drivers/video’ failed
make[1]: *** [drivers/video] Error 2
Makefile:1104: recipe for target ‘drivers’ failed
make: *** [drivers] Error 2
real 1m7.629s
user 2m27.900s
sys 0m20.864s
Make did not build
Retrying …
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CHK include/generated/bounds.h
CHK include/generated/timeconst.h
CHK include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
CHK scripts/mod/devicetable-offsets.h
CHK include/generated/compile.h
CHK kernel/config_data.h
CC drivers/video/tegra/dc/hdcp/hdmihdcp.o
/usr/src/kernel/nvidia/drivers/video/tegra/dc/hdcp/hdmihdcp.c: In function ‘link_integrity_check’:
/usr/src/kernel/nvidia/drivers/video/tegra/dc/hdcp/hdmihdcp.c:1940:16: error: unused variable ‘nonce’ [-Werror=unused-variable]
unsigned char nonce[HDCP_NONCE_SIZE];
^~~~~
cc1: all warnings being treated as errors
scripts/Makefile.build:335: recipe for target ‘drivers/video/tegra/dc/hdcp/hdmihdcp.o’ failed
make[4]: *** [drivers/video/tegra/dc/hdcp/hdmihdcp.o] Error 1
scripts/Makefile.build:649: recipe for target ‘drivers/video/tegra/dc’ failed
make[3]: *** [drivers/video/tegra/dc] Error 2
scripts/Makefile.build:649: recipe for target ‘drivers/video/tegra’ failed
make[2]: *** [drivers/video/tegra] Error 2
scripts/Makefile.build:649: recipe for target ‘drivers/video’ failed
make[1]: *** [drivers/video] Error 2
Makefile:1104: recipe for target ‘drivers’ failed
make: *** [drivers] Error 2
Make did not successfully build
Please fix issues and retry build
modinfo uvcvideo | grep “version:”
version: 1.1.1
srcversion: 36C804DD978CA18E839ECD1
I meet the same problem, have you solved it? Thanks for your reply.
I have solved it, you can refer to this issue: https://github.com/jetsonhacks/buildLibrealsense2Xavier/issues/18