Update
On April 2nd, Microsoft announced that they are consolidating the Kinect for Windows around a single sensor. They will discontinue “Kinect for Windows V2” described below. However the replacement will be a regular Xbox One Kinect Sensor Bar
Introduction
Open source drivers (OpenKinect libfreenect2) are available for the Kinect for Windows V2. Most installations are for PC based, x86 machines. The Jetson TK1, being an ARM V7 machine poses some special challenges for porting the source over. Most of those issues have been addressed, and an installation script is now available. Looky here:
http://youtu.be/URhu-fAUWWQ
Introduction
While most of the porting issues are straightforward, some special code needed to be written to get good performance out of the Kinect sensor on the Jetson. There are also some hardware issues that need to be addressed.
Fortunately Lingzhu Xiang added Tegra support for the Jetson to libfreenect2 which addressed the performance issues. The code is available on Github at: https://github.com/xlz/libfreenect2. Here is the “how to”: Jetson TK1 HOWTO. The Tegra code allows frame processing at 60Hz. Way excellent!
You might remember from an earlier post (Kinect V2 with libfreenect2) that there were issues using the built in USB controller. So a Syba 19-Pin USB 3.0 Header Mini PCI-Express Card with Female USB 3.0 Cable SD-MPE20142
Note: Please read the Built In USB below for notes on how to get this to work with the full size USB 3.0 connector on the Jetson.
Also, remember that in order to use the Kinect for Window V2 with the Jetson, the Kinect needs to be initialized with a Windows 8.1 machine. This should be done before plugging the Kinect into the Jetson for the first time.
This installation was run on a Jetson with L4T 21.2 with OpenCV and CUDA 6.5 installed.
(See JetPack installation). After the flash installation, the Post Flash Setup procedure was performed, which prepares the Jetson for desktop operation. After using JetPack and performing the post flash setup, the Jetson is prepared as the starting point for libfreenect2 installation.
Another note, the Jetson feature of USB autosuspend needs to be disabled for the Kinect V2 to work correctly. This is taken care of by the post flash setup by the installation of a startup script. WARNING: Replugging the Kinect may enable auto suspend again. To disable, execute this:
$ sudo sh -c ‘for dev in /sys/bus/usb/devices/*/power/autosuspend; do echo -1 >$dev; done’
You can check the result:
$ grep . /sys/bus/usb/devices/*/power/autosuspend
which should show all entries as -1.
Updates
Since the first article, there were a couple issues that have been addressed in xlz’s repository.
First, xlz reverted a previous libfreenect2 commit to fix MAX_ISO_BUFFER_LENGTH which was causing libusb issues.
Second, libfreenect2 installs libopencv-dev, which overwrites the CUDA accelerated opencv4tegra library. The install script on the How-To removes the libopencv-dev and adds libjpeg-turbo8-dev which installs the ‘turbojpeg.h’ header file.
Installation
The demo that I originally showed included a colormap depth image, which required changes to Protonect. I also wrote an installation script so that the installation process could be automated. I still have an updated fork of xlz’s repository, but there aren’t that many changes to it.
The installation script is available on Github:
Download it to the Jetson. In the video above, the script is placed into the Home (~/) directory.
$ sh installLibfreenect2.sh
This will install the code dependencies, download the libfreenect source and dependent libraries, and compile them.
Once the installation is complete, you can reboot the Jetson. This will enable the udev Kinect rules, which allows user access to the Kinect. If you do not reboot the Jetson, you will have to use the ‘sudo’ command before running the demo.
Demo
Switch to the demo directory:
$ cd libfreenect2/examples/protonect/bin
(This assumes that libfreenect2 was installed in your home directory). To run the demo:
$ ./Protonect
If all went well, you should see the depth and rgb images.
Built In USB
First, The mPCIe card mentioned above does not appear to have any issues, and seems to work reliably. If the Kinect V2 is plugged into the card’s port when the system boots, it works properly. The Kinect can also be hot plugged into the card’s port and it works properly.
The Jetson’s native USB 3.0 port works, but seems less reliable and requires a few extra steps.
Notes: The term ‘hub’ refers to an external, powered, USB 3.0 7 port hub connected to the full size USB connector on the Jetson. I am using a USB 3.0 hub with a keyboard and mouse also plugged into it, along with the Kinect. Before running Protonect each time, I checked the auto suspend state to make sure everything was -1. Replug means to unplug the device, wait a few seconds, and then plug it back in. Protonect is an example program which displays the RGB, infrared and depth camera streams.
FAILURE:
If the Jetson is booted with the hub and Kinect plugged into the hub, when Protonect is executed, it fails. Protonect gets to the point where it prints out ‘device firmware: 4.3.3912.0.7’ and gets deadlocked in waitForNewFrame. No windows appear., but the [TegraJpegRgbPacketProcessor] heartbeat message appears.The [CudaDepthPacketProcessor] heartbeat does NOT appear .
Subsequent attempts to run Protonect also fail. Protonect fails after it prints out the 92 bytes of raw data and then:
[code][CommandTransaction::receive] bulk transfer failed! libusb error -1: LIBUSB_ERROR_IO
Segmentation fault[/code]
This happens in the case where the Kinect remains in place, the case where the Kinect is replugged, and the case where the hub is replugged.
SUCCESS:
A) Boot the Jetson with the USB hub connected. Do not run Protonect.
B) Once the Jetson boots, replug the hub.
C) Turn off auto suspend
D) Run Protonect.
This brings up the program as expected.
The Kinect can be plugged into the hub at boot time, or after the replug. I did have issues in the case where the Jetson booted without the Kinect in the hub, unplug the hub, plug the Kinect into the hub, plug the hub back into the Jetson. In that case, the keyboard and mouse failed to be recognized. That could be a hub issue.
I played with the different combinations for several hours. The boot, replug the hub solution worked consistently, though there were times that replugging did not show the Kinect using “$ lsusb -t” and required replugging the hub once again.
Since the L4T kernel lags behind the mainstream Linux one, it’s possible that the above issue has been addressed. At this point, my curiosity has been more than satisfied, and I’m going to rate Kinect V2 support as working if still experimental. With the addition of xlz’s Tegra code, it is also at the point of being usable.
Here’s the unboxing and demo video for what it looks like when it’s up and running:
