Jetson Orin + RealSense in 5 minutes

The JetsonHacks Github repository offers kernel modules which allow you to install and run a RealSense camera in around 5 minutes. Looky here:

Background

The Intel RealSense cameras were some of the pioneers in affordable RGBD cameras. RGBD meaning Color + Depth. In addition to a pair of infrared cameras, RealSense Cameras like the D435i use an infrared laser projector to help illuminate indoor scenes. This allows for better performance in low light situations. The RealSense cameras have on board compute to help build the depth map and overlaying the RGB camera image for a point cloud.

For several different computers, the RealSense cameras offer a near plug and play experience using the RealSense SDK. For the Jetsons, it’s always been a little bit more trouble to run natively. There is a work around which allows you to use the RealSense SDK from user space on the Jetson. But there is a tradeoff in speed and accuracy which Intel warns against in a production environment.

Prior to JetPack 6, in order to run RealSense natively you had to patch and recompile the kernel and enable some kernel modules on the Jetson. Let’s say this was unfun, and the user had to know a little more than they should to get things running smoothly.

A nicer solution

With the advent of JetPack 6, the game changed. The important RealSense changes all happen in loadable modules, which means the user doesn’t need to build and install a new kernel. So as long as we have the new pre-built modules, the install is straight forward.

What you need

To get started, you’ll need:

1. Your Jetson; Jetson Orin Nano (Super) Developer Kit or Jetson AGX Orin Developer Kit
2. RealSense Camera, such as a RealSense D435i (Amazon Link)
3. An Internet connection

If you really want to build the kernel modules …

The JetsonHacks Github repository has a build directory which contains instructions on how to patch and build the modules. Follow the directions there, we also do a walk through in the video above.

5 Minute Install

The key to the 5 minute install is the pre-built kernel modules which enables you to install and use the RealSense SDK from a Debian package.

Read through the README file in JetsonHacks Github repository for step by step directions. The outline is:

1. Untar the modules
2. Make sure the checksums match
3. Switch to the install_modules directory
4. Run install-realsense-modules.sh
5. Register the Intel Librealsense public key
6. Add the server to your Ubuntu repositories
7. Install librealsense2-utils
8. Install librealsense2-dev
9. Run realsense-viewer to verify everything works
10. Update the camera firmware if needed

Camera Format Changes

The patches for the kernel modules add depth camera frame format information, and frame accurate metadata for a variety of functions, such as IMUs. The camera formats added are:

1. Greyscale 16 L/R (Y16I)
   • FourCC: V4L2_PIX_FMT_Y16I
   • Description: Interleaved 16-bit greyscale format, typically for left/right stereo data.
   • GUID: UVC_GUID_FORMAT_Y16I
2. Depth data 16-bit (D16)
   • FourCC: V4L2_PIX_FMT_Z16
   • Description: 16-bit depth data (mapped to the same pixel format as Z16 but with a distinct GUID).
   • GUID: UVC_GUID_FORMAT_D16
3. Packed raw data 10-bit (W10)
   • FourCC: V4L2_PIX_FMT_W10
   • Description: 10-bit packed raw data, formatted as 8888[2222] (likely 8-bit groups with 2-bit padding).
   • GUID: UVC_GUID_FORMAT_W10
4. Confidence data (C )
   • FourCC: V4L2_PIX_FMT_CONFIDENCE_MAP
   • Description: Packed confidence data (two pixels per byte, [44] format), often paired with depth data.
   • GUID: UVC_GUID_FORMAT_CONFIDENCE_MAP
5. Raw data 8-bit (RAW8)
   • FourCC: V4L2_PIX_FMT_GREY
   • Description: 8-bit monochrome raw data, often used in fisheye or IR cameras.
   • GUID: UVC_GUID_FORMAT_RAW8
6. Raw data 16-bit (RW16)
   • FourCC: V4L2_PIX_FMT_RW16
   • Description: 16-bit raw data (legacy format for backward compatibility).
   • GUID: UVC_GUID_FORMAT_RW16
7. 16-bit Bayer BGBG/GRGR
   • FourCC: V4L2_PIX_FMT_SBGGR16
   • Description: 16-bit Bayer pattern (BGGR ordering), typically for raw sensor data.
   • GUID: UVC_GUID_FORMAT_BAYER16
8. Z16 Huffman Compression (Z16H)
   • FourCC: V4L2_PIX_FMT_Z16H
   • Description: 16-bit depth data with custom Huffman compression.
   • GUID: UVC_GUID_FORMAT_Z16H
9. Frame Grabber (FG )
   • FourCC: V4L2_PIX_FMT_FG
   • Description: Format for frame grabber data (specific use case unclear, possibly diagnostic).
   • GUID: UVC_GUID_FORMAT_FG
10. SR300 Depth/Confidence (INZC)
    • FourCC: V4L2_PIX_FMT_INZC
    • Description: Planar format combining depth and confidence data, specific to SR300 cameras.
    • GUID: UVC_GUID_FORMAT_INZC
11. Relative IR (PAIR)
    • FourCC: V4L2_PIX_FMT_PAIR
    • Description: Relative infrared data, likely for stereo IR processing.
    • GUID: UVC_GUID_FORMAT_PAIR

These formats are added to support advanced imaging devices, such as Intel RealSense cameras (e.g., D435, SR300), which use VCSEL-based projectors for depth sensing.

Notes

In the video, the installation is being shown on a NVIDIA Jetson Orin Nano Super Developer Kit. The Intel RealSense D435i camera is attached to the Jetson.