Update
On April 2nd, Microsoft announced that they are consolidating the Kinect for Windows around a single sensor. They will discontinue “Kinect for Windows V2” described below. However the replacement will be a regular Xbox One Kinect Sensor Bar
along with a Kinect Adapter for Windows
which will provide a functionally equivalent solution. You will need both a Kinect Xbox One sensor and the adapter for use with the Jetson. The Kinect Adapter for Windows converts the output from the Kinect to USB 3.0. The advantage of this setup is that you can use the Kinect sensor from your Xbox One, or at least have an excuse to get a Xbox One + Kinect
for “research” purposes.
Introduction
In October, I purchased a Kinect for Windows V2 (a development version of the XBOX One Kinect) with the intent of connecting it to a NVIDIA Jetson TK1. It took a few months for me to get the Kinect up and running on the Jetson. Looky here:
Back in October, people over at OpenKinect were getting libfreenect2 up and running on PC based machines. Porting libfreenect2 over to the Jetson faced some challenges using this new library.
The Kinect V2 provides a JPEG stream 1920×1080 at 30 FPS, which is ~ 40MB/s, and an 16 bit 512×424 depth stream, and an 11 bit 512×424 infrared image stream. Lots of bits, coming in pretty quick. Therefore the Kinect V2 requires USB 3.0 SuperSpeed mode. Remember that the SuperSpeed mode of USB 3.0 is around 5 Gb/s. The library libusb is used for interfacing to USB, but was reporting having some issues with SuperSpeed on chipsets other than Renesas.
The Jetson does not use Renesas for it’s USB 3.0 chip. Fortunately, there are Mini PCI-Express Cards with that chipset. I bought this one: Syba 19-Pin USB 3.0 Header Mini PCI-Express Card with Female USB 3.0 Cable SD-MPE20142
. Note: It is a full size Mini PCI-Express card, so it does hang over the edge of the Jetson when installed. Also, the card requires a floppy drive power connector. I found one at a local Frys electronics store that hooked into the Jetson onboard Molex power connector.
At the time, LT4 19.3 also had USB issues which needed to be sorted out before libfreenect2 could work. The good news is that a couple of months later LT4 21.2 appeared which addressed the USB issues that I was experiencing. By late January, libfreenect2 with an example program named Protonect was up and running on the Jetson. However, the JPEG stream from the Kinect was being decoded on the CPU which led to slow frame rates. At that point, I was getting ready to write GPU code to speed that process up a bit.
Here’s where we all got lucky. It turns out that Lingzhu Xiang was working on adding Tegra support for the Jetson to libfreenect2. The code is available on Github at: https://github.com/xlz/libfreenect2. Here the “how to”: Jetson TK1 HOWTO.
The JPEG decoder is derived from GSTJpeg, and the depth image decoder is written in CUDA code. The decoders work at over 50 fps on the Jetson when the images are being displayed, greater than 60 fps when not being displayed. Sweet!
Note
In late January after LT4 21.2 came out, libfreenect2 worked with both the built in USB 3.0 and the PCI Express card previously mentioned. I was unable to get the built in USB 3.0 to work in mid-February with libfreenect2, although I was able to get the PCI Express card to work correctly, as seen in the demo.
These issues have been resolved or work arounds have been figured out. Read about how to install Kinect V2 and libfreenect2 on the Jetson here: Install Kinect – Part II
Another Note
In order to initialize the Kinect for Windows V2, you will need access to a Windows 8.1 machine to register the device and get it running. I happen to use a Dell Inspiron 3000 Series i3847-3850BK Desktop (3.5 GHz Intel Core i3-4150 Processor, 8GB DDR3, 1TB HDD, Windows 8.1)
. The Dell is setup as a dual boot machine with Ubuntu 14.04 which is what I use as a host machine to flash the Jetson boards.
2 Responses