NOTE: There is an updated article for use with L4T 28.1 (JetPack 3.1) and above. Please see: Intel RealSense Camera librealsense – NVIDIA Jetson TX Dev Kits
Intel RealSense cameras can use an open source library called librealsense as a driver for the Jetson TX2 development kit. Looky here:
tx
Background
Note: This article is intended for intermediate users who are comfortable with Linux kernel development, and can read and modify simple shell scripts if needed.
In earlier articles we talked about the Intel RealSense R200 Camera, which is a relatively inexpensive RGBD device in a compact package. The camera uses USB 3.0 to communicate with a computer.
Intel has made available an open source library, librealsense on Github. librealsense is a cross platform library which allows developers to interface with the RealSense family of cameras, including the R200. Support is provided for Windows, Macintosh, and Linux.
There are two major parts to getting the R200 camera to work with the Jetson. First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.
A kernel module is code that can be accessed from the kernel on demand, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems.
A module is compiled code which is stored as a file separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to modify the entire kernel. We will be building a module called uvcvideo to help interface with the RealSense camera. Normally uvcvideo is built-in to the kernel, we will designate it as a module as part of our modification. We will modify uvcvideo to recognize the RealSense camera data formats.
The second part of getting the R200 to work with the Jetson TX2 is to build and install librealsense.
Kernel and Module Building
Note: In the video above, the installation was performed on a newly flashed L4T 27.1 TX2 using JetPack 3.0. The kernel sources were downloaded and built before shooting the video.
In the previous article, we built a new kernel for the Jetson TX2. Please refer to the article for the details. Once completed, you’re ready to install librealsense..
Install librealsense
A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.
$ cd $HOME
$ git clone https://github.com/jetsonhacks/installLibrealsenseTX2.git
$ cd installLibrealsenseTX2
$ ./installLibrealsense.sh
This will build the librealsense library and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and the camera can be accessed from user space.
USB Video Class Module
Note: This step assumes that the kernel sources are located in /usr/src/kernel and that the kernel is to be installed on the board. If this is not your intent, modify the script accordingly. applyUVCPatch.sh has the command to patch the UVC driver with the RealSense camera formats.
The third major step is to build the USB Video Class (UVC) driver as a kernel module. This is can be done using the script:
$ ./buildPatchedKernel.sh
The buildPatchedKernel script will modify the kernel .config file to indicated that the UVC driver should be built as a module. Next, the script patches the UVC driver to recognize the RealSense camera formats. Finally the script builds the kernel, builds the kernel modules, installs the modules and then copies the kernel image to the /boot directory.
Note: The kernel and modules should have already been compiled once before performing this step.
One more minor point of bookkeeping. In order to save power, the Jetson TX2 will auto-suspend USB ports when not in use for devices like web cams. This confuses the RealSense camera. In order to turn auto-suspend off, run the following script:
$ ./setupTX1.sh
Once finished, reboot the machine for the new kernel and modules to be loaded.
Examples are located in the directory /librealsense/build/examples
Note: If you intend on using the RealSense camera with Robot Operating System (ROS), you may find this article useful: Intel RealSense Package for ROS on NVIDIA Jetson TX2
Conclusion
So there you have it. This has been a little bit more involved than some of our other projects here, but if you are interested in this kind of device, well worth it.
Notes
There are several notes on this project:
- In the video above, the installation was done on a Jetson TX2 running L4T 27.1 immediately after being flashed by JetPack 3.0
- QtCreator and Qt 5 are installed as dependencies in the librealsense part of the install. There are QtCreator project files located in the librealsense.qt directory. The project files build the library and example files. If you do not use QtCreator, consider modifying the installer script to take QtCreator out.
- The librealsense examples are located in librealsense/build/examples after everything has been built.
- These scripts install librealsense version v1.12.1 (last commit 7332ecadc057552c178addd577d24a2756f8789a)
- The RealSense R200 is the only camera tested at this time.
- Issues have been reported on the Jetson TX2 if the camera is plugged directly into the USB 3.0 port. The workaround is to use a powered hub. It’s unknown at this time if the TX2 suffers from the same issue.
- The RealSense cameras are USB 3.0 devices. If the camera is attached to a USB hub, the USB hub must support USB 3.0
- Intel RealSense Stereoscopic Depth Cameras is a comprehensive overview of the stereoscopic Intel RealSense RGBD imaging systems
44 Responses
This article and code is insanely useful. Thank you!
From a fresh TX2 with JetPack 3.0, L4T 27, Ubuntu 16.04 host/TX2, full install of pkgs like cuda, opencv etc. I went through the instructions to
-Build TX2 Kernel
-Install librealsense
-Install ROS
It may be helpful to also link to the Realsense ROS article here, because I missed that as the 4th step originally. The resulting error message about uvcvideo input/output error did not make it obvious that I needed to install realsense for ROS, but that was the solution.
I got 2 R200s running on my TX2 simultaneously with ROS, using those 4 articles.
Done, and done! I hope that when you start doing something clever with the cameras (and I know you will!) you can come back and share it with us.
My primary purpose for this setup is playing with drones. For the video that you did outside, showing guys waving arms and such, was that done during the day or the night? At what kind of range could you make the point cloud work?
Lastly, could you use the IR cameras to actually get IR pictures and that you could derive temperatures from? That would be very cool.
Mine are on order and are supposed to be here within a couple of weeks.
Truly fascinating stuff!
I don’t remember doing a RealSense Camera video outside. Someone else perhaps?
You can get the IR images from libRealsense , you have to do the calculations to convert them to temperature information.
Hey, thanks for the article. Would be nice if you could change the buildPatchKernel to take both “Y” and “y”, I could not understand why it kept saying kernel changes canceled until I opened the source and found out it had to be capital xD
That’s a good idea. You should submit a pull request on Github. Thanks for reading!
Are all your outstanding scripts designed to be run on the host or, for example, does installing librealsense occur only on the TX2? Does librealsense work on the host (in Ubuntu?)
Most of the scripts here are for running on the Jetson Dev Kits. librealsense has instructions for building/installing on the host pc. Thanks for reading!
Hi,
we are using the SR300 and your guide is sadly not working. We don’t get the RGB image and the IR image looks wrong. We got the SR300 working on TX1 but just can’t seem to get it working on TX2. Any idea what could be wrong? We would love a guide for the SR300 🙂
Great tutorials!
Everything went well, but we have also problems with SR300, depth image is flickering and jumping. Have you ever had a similar issue?
I have not used a SR300. The video formats between the R200 and SR300 may be different. In the librealsense repository, examine the script:
https://github.com/IntelRealSense/librealsense/blob/master/scripts/0002-LR200-ZR300-and-SR300-Pixel-Formats.patch
and compare that with the formats that were patched during the build.
Right, the problem should be here. When comparing, the difference between your patch and 0001+0002 patches is not big, only 2 lines are different (mainly in Depth:IR 16:10 26-bit (INZI)). However, after applying the modified patch, the IR/depth images are still jumping (even if applying patch went well and no errors during compilation).
You should contact NVIDIA. My next guess would be that there is a bug in the USB stack that would be difficult to find if it is in the USB firmware.
I have the same problem, SR300 works on TX1 but not on TX2. Did anybody find a solution?
Hi! I encounter the same problem with SR300 on TX2. The installation went well, but the depth image is flickering and the infrared image is not working at all. Did anybody find a solution by now?
Unfortunately I do not have a SR300, so it’s difficult to help. Are you using the same USB hub as you did with the TX1? That seems to be one of the issues that you can run into.
There is some kind of bug in the usb stack as seen in the logs when using a realsense R200:
[ 81.899869] USB Video Class driver (1.1.1)
[ 89.644529] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 5
[ 89.651540] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_DEC_SSPI_CLOCK request
[ 112.790658] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 4
[ 112.797636] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_INC_SSPI_CLOCK request
We’ll have to see if this issue is addressed in the new JetPack 3.1 release coming out the week of July 24, 2017.
Man, I sure hope SO!!!!!
I noticed that the kernel sources of both TX1 and TX2 in JetPack v3.1 already contain some of the necessary camera formats for RealSense devices. I was able to get librealsense to work right out of the box, without patching the kernel.
A small modification in librealsense is needed though, I’ve posted it in the link below.
https://github.com/IntelRealSense/librealsense/issues/563
This is certainly good news! The process noted here is much too difficult for casual users. Thanks for the heads up, I’ll have to update the article once I get everything working.
So to clarify, am I right in thinking that Jetpack3.1 on TX2 will already recognise the R200 camera? But to also have the examples working requires a rebuild of the kernel (as per previous article), but no further patches are required?
Now on my second day on this, but determined to get it working! My experience is that the kernel build goes well but I get an error ‘Hunk#1 failed at 621’ when I try to apply the patch as above. This might explain why, of course.
As always, a thousand thanks to those who contribute!
Hi Mike,
My current understanding is that no modifications to the kernel are needed.
There is a patch that is needed so that librealsense recognizes the internal UVC module: https://github.com/IntelRealSense/librealsense/pull/442/files
Another difference between newer versions of librealsense and this version is that newer versions use CMake. You should be able to follow the librealsense Ubuntu installation directions and get everything working fairly easily once you apply the patch in the librealsense library code. Good luck!
Great. So for info on a completely fresh install of Jetpack 3.1, dmesg shows the camera is detected:
[ 7793.739594] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 7793.746786] usb 2-1: Product: Intel RealSense 3D Camera R200
[ 7793.752496] usb 2-1: Manufacturer: Intel Corp
I’m pretty new to working with Ubuntu in this way, but I’ll try to do the patch and go from there My main focus is to be able to get on with using the R200 from within ROS… ho hum!
Your advice has been fantastic – thanks.
So I started with a fresh install of jetpack 3.1 and modified the librealsense code as in pull request 442 above. lsmod lists uvcvideo but examples give an error ‘kernel module not loaded’. I tried again but with a rebuilt kernel with acm and ch341 as modules as per your recent video. Same result.
I think I have reached the limit of my expertise – I feel so close yet so far still!! I’ll have to wait now and hope you will do another of your marvellous videos!
Cheers, Mike
I played with it last night, the patch isn’t quite right for the TX2. I’ll be updating the repository, but if you want to play with it sooner all you have to do is delete the part where it’s checking for the uvcvideo module. You know it is built into the kernel, so no need to check for it. An alternative is to comment out the place where it throws the exception.
WOW! I ran the script in the repository you worked on last night and now my camera works!… so many thanks.
Now onwards with ROS…
Cheers!
Glad you got it to work. Good luck on the next step, it should “just work”.
I am not able to find . buildPatchedKernel.sh file on librealsense git repository ,can I neglect that step? If no what should I do?
The repository has been updated to L4T 28.1. If you are still using an earlier version, checkout the Tag version for that operating system.
Hi, thanks for your amazing video.
I just flash my TX2 with JetPack3.1 . When I connect the RealSense R200 it seems it isn’t recognized. I Doesn’t appear in lsusb and dmesg output. Is it possible that ovcvideo module/part of kernel is not installed properly?
the following output is making me suspicious:
dmesg | grep uvcvideo
[ 6936.890131] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
[ 6936.897746] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
[ 6936.897748] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
[ 7794.114136] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
[ 8394.921380] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
[ 8394.931965] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
[ 8394.943546] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
If I try to run example available in build/example folder, I got the following error:
what(): uvcvideo kernel module is not loaded
I am trying to understand if it is a kernel problem or a patch problem or what else. Any advise?
PS: camera is working on other ubuntu machines
Which instructions did you follow in installing librealsense? Did you rebuild the kernel?
I flash with jetpack 3.1, than I build the kernel with your tutorial without modifications to config file. Went through the 3 steps and reboot.
There is a note in the first line of this article. It reads: Note: This article has been updated for use with L4T 28.1+. Please see: Intel RealSense Camera librealsense – NVIDIA Jetson TX Dev Kits
You appeared to find this confusing. How can I reword it so that you would go to the new article instead of following the directions in this article?
The directions in this article are for L4T 27.1 (JetPack 3.0). For L4T 28.1 and above (JetPack 3.1), please refer to the article: https://jetsonhacks.com/2017/08/14/intel-realsense-camera-librealsense-nvidia-jetson-tx-dev-kits/ You should not need to recompile the kernel in order to get the R200 working with the Jetson TX2.
Thanks for the support. As I said, it is 3 days I’ve been working and I think I tried all possible way. My first attempt was to flash the TX2 and then install the realsense lib for TX2. Can someone confirm if the uvcvideo module should appear in the lsmod output of a flashed TX2?
Hi Laupl0082,
The RealSense R200 is a USB 3.0 device. First, lsusb should indicate that the device is present:
ID 8086:0aB0 Intel Corp
If the R200 is detected, then uvcvideo will show in:
$ lsmod
uvcvideo
My guess is that you are using a USB 2.0 hub. If the R200 does not show up in lsusb, then it does not load the uvcvideo module.
same USB HUB and R200 connected to the laptop works. So I think I can exclude this as a possible cause.
Is the USB hub 3.0? It should have blue plugs, not white or black.
Yes, USB 3.0 HUB. all blue plugs
What does:
$ dmsg –follow
show when you unplug the R200, and then plug it back in?
In your case, does the R200 show up in:
$ lsusb
lsusb doesn’t show anything about the R200 when I plug it in. No changes between plugged and unplugged.
the last row in dmesg is the following:
[ 6936.890131] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
It appears as if your kernel does not match your uvcvideo module. Is this a modified kernel?
There are two issues:
1) 4.4.83 indicates that you have the wrong kernel number in you module, it should be 4.4.38
2) There is no local version number set on your uvcvideo module, it should match the kernel you are using (e.g. -tegra)
I would suggest that you flash the Jetson with JetPack 3.1. Plug the camera in and check to see if it shows up in lsusb, and if the uvcvideo module is loaded. Then you can install librealsense following the instructions on: https://jetsonhacks.com/2017/08/14/intel-realsense-camera-librealsense-nvidia-jetson-tx-dev-kits/
Please report back after you do this.
H,
I followed all instructions building the kernel (buildLibrealsenseTX2), installing librealsenseTX (installLibrealsenseTX2) and installRealSense2ROS. When I try to catkin_make there is an error……
/opt/ros/kinetic/include/librealsense2/hpp/rs_frame.hpp:1131:13: error: no match for call to ‘(rs2::asynchronous_syncer) (rs2::frame)’
on_frame_function(frame{ fref });
^
is there a conflict with new libraries from intel – should other branches than master from realsense be used?
System is Jetson TX2 with Jetpack 3.3
No idea why this happens – any help is highly appreciated.
Cheers
Uli
The librealsense libraries and the ROS wrapper are tied together. The ROS wrapper is for a specific release of librealsense. Typically you would find the matching releases in each of the JetsonHacks Github repository under ‘Releases’ and use those.
Lately Intel has recommended using the debian release of librealsense on the Jetsons which uses the UVC backend. You can check out: https://github.com/jetsonhacks/installRealSenseSDK