This article has been superseded. Please see: Install Kinect V2 – Part II
The article below is for historical purposes only, and does not reflect the current state of the project.
Open source drivers (OpenKinect libfreenect2) are available for the Kinect for Windows V2. Most installations are for PC based, x86 machines. The Jetson TK1, being an ARM V7 machine poses some special challenges for porting the source over. Most of those issues have been addressed, and an installation script is now available. Looky here:
Introduction
While most of the porting issues are straightforward, some special code needed to be written to get good performance out of the Kinect sensor on the Jetson. There are also some hardware issues that need to be addressed.
Fortunately Lingzhu Xiang added Tegra support for the Jetson to libfreenect2 which addressed the performance issues. The code is available on Github at: https://github.com/xlz/libfreenect2. Here is the “how to”: Jetson TK1 HOWTO. The Tegra code allows frame processing at 60Hz. Way excellent!
You might remember from an earlier post (Kinect V2 with libfreenect2) that there were issues using the built in USB controller. So a Syba 19-Pin USB 3.0 Header Mini PCI-Express Card with Female USB 3.0 Cable SD-MPE20142
was acquired. Note: It is a full size Mini PCI-Express card, so it does hang over the edge of the Jetson when installed. Also, the card requires a floppy drive power connector. I found one at a local Frys electronics store that hooked into the Jetson onboard Molex power connector.
Also, remember that in order to use the Kinect for Window V2 with the Jetson, the Kinect needs to be initialized with a Windows 8.1 machine.
This installation was run on a Jetson with L4T 21.2 with OpenCV and CUDA 6.5 installed.
(See JetPack installation). After the flash installation, the Post Flash Setup procedure was performed, which prepares the Jetson for desktop operation. After using JetPack and performing the post flash setup, the Jetson is prepared as the starting point for libfreenect installation.
Another note, the Jetson feature of USB autosuspend needs to be disabled for the Kinect V2 to work correctly. This is taken care of by the post flash setup mentioned above.
Issues, Issues, Issues …
When I tried to install Lingzhu Xiang’s code, I ran into several issues. The installation script below to install libfreenect2 addresses these issues.
The first issue is that the original installation recommended installing opencv-dev. Unfortunately this overwrites the hardware accelerated Tegra OpenCV library that JetPack installs. This was easy enough to rectify, the only real issue was that the include file turbojpeg.h was missing. I built a little repository for the installation that was forked from Lingzhu Xiang’s and added turbojpeg.h to it. Also, the link to libturbojpeg.so needed to be moved around a little.
The second issue is with libusb. It appears as if libusb/libfreenect2 was changed after the original repository was published. In this installer, there is a patch included for the libusb headers (usbfs.h) which sets the MAX_ISO_BUFFER_LENGTH to be 49152.
Note: Earlier versions of libfreenect2 worked with the built in USB 3.0 of the Jetson, as Lingzhu Xiang reported. The current version (even with the patch) does not. This is an outstanding issue.
By adding the file 90-kinect2.rules to the udev permissions, Kinect V2 is given permission to run in user space. This is included in the script setup.
For the demo, a colormap of the depth map is added. By pressing one of the number keys, different colormaps can be used.
Note: The NVIDIA Gstjpeg headers are needed to enable hardware acceleration and JPEG decoding. These headers are downloaded using the installation script and placed into the libfreenect/depends directory.
Installation
The installation script is available on Github:
Download it to the Jetson. In the video above, the script is placed into the Home (~/) directory.
$ sh installLibfreenect2.sh
This will install the code dependencies, download the libfreenect source and dependent libraries, and compile them.
Once the installation is complete, you can reboot the Jetson. This will enable the udev Kinect rules, which allows user access to the Kinect. If you do not reboot the Jetson, you will have to use the ‘sudo’ command before running the demo.
Demo
Switch to the demo directory:
$ cd libfreenect2/examples/protonect/bin
(This assumes that libfreenect2 was installed in your home directory). To run the demo:
$ ./Protonect
If all went well, you should see the depth and rgb images.
Good luck!
32 Responses
hey,
The information was really helpful. Are you working on multiple kinect implementation?
CAn you help me out with that?
Hi Vedanshu,
I am not currently working on a multiple Kinect V2 implementation. Do you have an application in mind?
Hi JetsonHacks,
When I ran sh installLibfreenect2.sh, it reached a fatal error at 47%. It shows:
[ 47%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/src/./cuda_compile_generated_cuda_depth_packet_processor.cu.o
/home/sebastian/gist1bd2830cc1a5790a6ac2-3d786bbb39cd936a5e823a13bea703817447c2fb/libfreenect2/examples/protonect/src/cuda_depth_packet_processor.cu:29:25: fatal error: helper_math.h: No such file or directory
#include
Do you know what the problem is?
What version of L4T are you running, and which version of CUDA is installed?
I don’t know either of those things. How do I find out?
For CUDA, from a terminal execute:
$ nvcc –version
For L4T it’s a little bit tricker, it is in
/etc/nv_tegra_release
For CUDA, I tried the command and it said command not found. For L4T, I could not find the file in the place that you listed. Do I have to download it?
Are you running these commands on a NVIDIA Jetson TK1?
What is the result of:
$ uname -r
I am not. Do I need to be running it on NVIDIA Jetson TK1?
The instructions and script file are for installation on a NVIDIA Jetson TK1. You will likely get better results by following the install instructions for your particular machine on the OpenKinect/libfreenect2 repository on Github: https://github.com/OpenKinect/libfreenect2
Good luck!
Thank you for helping me. I will check out that site.
Hi.
Are there any guides or tutorials for libfreenect2?
The official libfreenect2 Github account is at: https://github.com/OpenKinect/libfreenect2
The official OpenKinect page: http://www.openkinect.org
is community portal where they have more information about the project.
I checked both, and neither had any guides. Do you happen to know any other sites?
No, I don’t know of anything beyond those two. Sorry.
Thank you for the links.
Now I trying that run libfreenect2 on Jetson TK1
and, there are some issues.
If you know, please some adbises.
————————————————
1.I tried to adjust CPU clock and GPU clock
# cat /sys/kernel/cluster/active
G
# cat /sys/kernel/debug/clock/gbus/rate
852000000
————————————————
2.And I use a mini PCI-Express USB3.0 19pin (Renesas nPD720200)
Power is on mora 4Pin on Jetson.
After some tried, I used external power supply, but the same issue is happening
————————————————
3.Result of lsusb
$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 045e:02c4 Microsoft Corp.
Bus 002 Device 002: ID 045e:02d9 Microsoft Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 045e:02d9 Microsoft Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
————————————————
4. I have installed them as described below.
https://gist.github.com/jetsonhacks/1bd2830cc1a5790a6ac2#file-installlibfreenect2-sh
That .sh file is as described below
sudo apt-get install -y build-essential libturbojpeg libtool autoconf libudev-dev cmake mesa-common-dev freeglut3-dev libxrandr-dev doxygen libxi-dev libjpeg-turbo8-dev
git clone https://github.com/jetsonhacks/libfreenect2.git
wget http://developer.download.nvidia.com/mobile/tegra/l4t/r21.2.0/sources/gstjpeg_src.tbz2
tar -xvf gstjpeg_src.tbz2 gstjpeg_src/nv_headers
mv gstjpeg_src/nv_headers/ libfreenect2/depends/
rmdir gstjpeg_src/
cd libfreenect2/depends
sh install_ubuntu.sh
sudo ln -s /usr/lib/arm-linux-gnueabihf/libturbojpeg.so.0.0.0 /usr/lib/arm-linux-gnueabihf/libturbojpeg.so
cd ../examples/protonect/
cmake CMakeLists.txt
make && sudo make install
cd ../..
sudo cp extras/90-kinect2.rules /etc/udev/rules.d/90-kinect2.rules
/bin/echo -e “\e[1;32mFinished.\e[0m”
————————————————-
The following message is displayed when I start the Protonect.
————————————————
$ sudo ./bin/Protonect
[sudo] password for ubuntu:
modprobe: FATAL: Module nvidia not found.
[CudaDepthPacketProcessorKernel::initDevice] device 0: GK20A @ 852MHz Memory 1892MB
[CudaDepthPacketProcessorKernel::initDevice] selected device 0
[Freenect2Impl] enumerating devices…
[Freenect2Impl] 8 usb devices connected
[Freenect2Impl] found valid Kinect v2 @2:3 with serial 017234451747
[Freenect2Impl] found 1 devices
[Freenect2DeviceImpl] opening…
[Freenect2DeviceImpl] opened
[Freenect2DeviceImpl] starting…
[Freenect2DeviceImpl] ReadData0x14 response
92 bytes of raw data
0x0000: 00 00 15 00 00 00 00 00 01 00 00 00 43 c1 1f 41 …………C..A
0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
0x0020: 0a 21 33 55 20 00 17 ba 00 08 00 00 10 00 00 00 .!3U ………..
0x0030: 00 01 00 00 00 10 00 00 00 00 80 00 01 00 00 00 ……….�…..
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
0x0050: 00 00 00 00 00 00 00 00 07 00 00 00 …………
==== CuDepthPakePro CommandResponse ==== 1302882
[Freenect2DeviceImpl] ReadStatus0x090000 response
4 bytes of raw data
0x0000: 00 22 00 00 .”..
[Freenect2DeviceImpl] ReadStatus0x090000 response
4 bytes of raw data
0x0000: 00 22 00 00 .”..
[Freenect2DeviceImpl] enabling usb transfer submission…
[Freenect2DeviceImpl] submitting usb transfers…
[Freenect2DeviceImpl] started
device serial: 017234451747
device firmware: 2.3.3913.0.7
[TegraJpegRgbPacketProcessor] avg. time: 15.8725ms -> ~63.0022Hz
[RgbPacketStreamParser::handleNewData] skipping rgb packet!
[RgbPacketStreamParser::handleNewData] skipping rgb packet!
[TegraJpegRgbPacketProcessor] avg. time: 15.9096ms -> ~62.8553Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1333ms -> ~66.0795Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1105ms -> ~66.1793Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1506ms -> ~66.0038Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1426ms -> ~66.039Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1493ms -> ~66.0098Hz
[TegraJpegRgbPacketProcessor] avg. time: 14.9997ms -> ~66.6679Hz
[TegraJpegRgbPacketProcessor] avg. time: 14.4716ms -> ~69.101Hz
[TegraJpegRgbPacketProcessor] avg. time: 14.429ms -> ~69.3047Hz
[RgbPacketStreamParser::handleNewData] skipping rgb packet!
[TegraJpegRgbPacketProcessor] avg. time: 14.908ms -> ~67.0779Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.0251ms -> ~66.5553Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1235ms -> ~66.1221Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1282ms -> ~66.1015Hz
[TegraJpegRgbPacketProcessor] avg. time: 15.1414ms -> ~66.0441Hz
What version of L4T are you running?
Is USB 3.0 enabled?
Have you turned off the USB autosuspend?
have you solve the problem,i have met the same situation too.thx
Hi, Thanks for great tutorials, did you have any success for extracting audio from Kinect V2 using Jetson Tk1?
Best
I believe that the current version, libfreenect2 0.2, does *not* support audio. From what I understand, people have been working on it, but most people are interested just in the video/depth so there hasn’t been a lot of progress in this area. Thanks for reading!
Hello, I installed the libfreenect2 but ~/libfreenect2/examples/protonect/bin folder does not exist after the installation. I installed the Jetpack 3.0 and the grinch kernel
in my TK1.
When you ran make, what were the error messages?
CMake Error at CMakeLists.txt:115 (MESSAGE):
Did not find Tegra JPEG library
— Configuring incomplete, errors occurred!
See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeOutput.log”.
See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeError.log”.
ubuntu@tegra-ubuntu:~/libfreenect2/examples/protonect$ make
make: *** No targets specified and no makefile found. Stop.
The complete messages when i run cmake:
ubuntu@tegra-ubuntu:~/libfreenect2/examples/protonect$ cmake CMakeLists.txt
— The C compiler identification is GNU 4.8.4
— The CXX compiler identification is GNU 4.8.4
— Check for working C compiler: /usr/bin/cc
— Check for working C compiler: /usr/bin/cc — works
— Detecting C compiler ABI info
— Detecting C compiler ABI info – done
— Check for working CXX compiler: /usr/bin/c++
— Check for working CXX compiler: /usr/bin/c++ — works
— Detecting CXX compiler ABI info
— Detecting CXX compiler ABI info – done
— Performing Test LIBFREENECT2_THREADING_STDLIB
— Performing Test LIBFREENECT2_THREADING_STDLIB – Failed
— using tinythread as threading library
— Could NOT find OpenCL (missing: OPENCL_INCLUDE_DIRS)
— Found CUDA: /usr/local/cuda-6.5 (found version “6.5”)
— Looking for XOpenDisplay in /usr/lib/arm-linux-gnueabihf/libX11.so;/usr/lib/arm-linux-gnueabihf/libXext.so
— Looking for XOpenDisplay in /usr/lib/arm-linux-gnueabihf/libX11.so;/usr/lib/arm-linux-gnueabihf/libXext.so – found
— Looking for gethostbyname
— Looking for gethostbyname – found
— Looking for connect
— Looking for connect – found
— Looking for remove
— Looking for remove – found
— Looking for shmat
— Looking for shmat – found
— Looking for IceConnectionNumber in ICE
— Looking for IceConnectionNumber in ICE – found
— Found X11: /usr/lib/arm-linux-gnueabihf/libX11.so
— Found OpenGL: /usr/lib/arm-linux-gnueabihf/libGL.so
— Looking for include file pthread.h
— Looking for include file pthread.h – found
— Looking for pthread_create
— Looking for pthread_create – not found
— Looking for pthread_create in pthreads
— Looking for pthread_create in pthreads – not found
— Looking for pthread_create in pthread
— Looking for pthread_create in pthread – found
— Found Threads: TRUE
— Using X11 for window creation
— Using GLX for context creation
— Looking for glXGetProcAddress
— Looking for glXGetProcAddress – found
— Looking for glXGetProcAddressARB
— Looking for glXGetProcAddressARB – found
— Looking for glXGetProcAddressEXT
— Looking for glXGetProcAddressEXT – not found
CMake Error at CMakeLists.txt:115 (MESSAGE):
Did not find Tegra JPEG library
— Configuring incomplete, errors occurred!
See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeOutput.log”.
See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeError.log”.
Hi Marco,
It is not clear what steps you took to get to the issue. Are you following the steps here: https://jetsonhacks.com/2015/03/03/install-kinect-v2-part-ii/
This article has been deprecated, and replaced with the above referenced link.
Yes, I follow the video instructions, but does not work and I try to run every line from the git.
Hi Marco,
You should be running a variation of the script located on:
https://jetsonhacks.com/2015/03/03/install-kinect-v2-part-ii/
Since these articles are a few years old, you will have to change the source files to reflect the L4T version that you have installed. It could be that the package simply does not work with versions of L4T newer than the one shown here.
Thank you so much for this great tutorial!
Would love to use NiTE-2.0.0 for Skeleton Tracking and running into some compiling issues. It doesn’t seem as straight forward to get NiTE-2.0.0 working and been navigating many compile errors. Would love to see a tutorial on this sometime… especially before Halloween. Got some Skeletons to track 🙂 Many thanks for all you do!
My understanding is that NITE is a x86 binary library and not available for ARM. Is the source code now available somewhere?
Hi,
At first thanks for the tutorial. I have a little problem when i use your scrip. In the step when you use cmake CMakeLists.txt, appear this:
CMake Error at CMakeLists.txt:115 (MESSAGE)
did not find Tegra JPEG library
–configuring incomplete, errors occurred!
I use Jetson TK1, and is a new installation using JetPack3.1 and I don’t know how fix this problem.
Thanks for all(my English is so bad, i know)
Thx very for this tutorial. I have get for 2 days one Jetson Nano 2GB and one Jetson nano 4GB and I have a Kinect V2. Its clear that I have try out to connect them. I use Jetpack 4.6. And have succefull compiled libfreenect2. I can start it with sudo ./Protonect cuda -nodepth and that run on both. If I start it with sudo ./Protonect cuda -norgb, than I see for a short time the depth image and afterwards I see several USB errors and the system is freezed. The complete USB communication is broken. Has here someone a hint, what could be the reason. I would like to build up a 3D scanner and without the kinect v2 it is hard to scann.