Connecting a Microsoft Kinect to NVIDIA Jetson TK1
The video describes connecting a Microsoft Kinect to the NVIDIA Jetson TK1 Development Kit.
The Kinect provides 3D depth information about a scene. This is commonly referred to as a RGBD device because the depth information is returned encoded in an RGB image, with the color of each pixel representing how far away an object is from the sensor. Red is close, green is in the middle, blue is furthest away.
The demonstration shows a Microsoft Kinect 360 model 1414, the original XBOX 360 sensor. The Jetson and the Kinect are plugged into a 7 port USB hub, along with a keyboard and a mouse.
Some more information about Kinect, here’s a link to Kinect on Wikipedia
Installing the OpenKinect Drivers
Installing the OpenKinect libraries (libfreenect) is straightforward. A prerequisite for this installation is Internet access and installation of the Universe package repositories. These commands are entered using the ‘Terminal’ app, which is commonly referred to as the command line.
Universe packages can be added with:
$ sudo apt-add-repository universe
$ sudo apt-get update
To build the freenect library, first make sure that you have the build tools and library dependencies installed:
$ sudo apt-get install cmake freeglut3-dev pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev
Then Git clone the libfreenect repository:
$ git clone git://github.com/OpenKinect/libfreenect.git
If you do not have Git installed, you will receive instructions on how to install Git at the command prompt.
As an alternative, you can get the libfreenect source from OpenKinect on Github
Build libfreenect
To build the libfreenect library, start in the Home directory at the command line. Then enter the following commands:
$ cd libfreenect
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
NOTE: If you would like to uninstall libfreenect at some later time, go the ‘build’ directory:
$ sudo make uninstall
USB Autosuspend
As noted in the video, there is an issue with the default setting for USB ports being turned off when not in use. The Jetson does this to save energy. If this happens when the Kinect is connected, the Jetson will not recognize the Kinect. A solution is to execute commands automatically on every boot-up that alter this behavior. You can put your commands into the “/etc/rc.local” boot up script. For example, run this on your board:
$ sudo gedit /etc/rc.local
Then add this near the bottom of the file but before the “exit” line:
# Disable USB auto-suspend, since it disconnects some devices such as webcams on Jetson TK1.
echo -1 > /sys/module/usbcore/parameters/autosuspend
Save the file.
You then reboot the machine so that the changes take effect.
$ sudo reboot
NOTE: If you would like to do this more selectively, you can enter this command on the command line:
sudo bash -c ‘echo -1 > /sys/module/usbcore/parameters/autosuspend’
which will cause autosuspend to be turned off until the next boot cycle.
Sample Programs
Once the Jetson powers back up, and the Desktop appears on the screen. Plug in the Kinect to the USB hub. NOTE:You may have to unplug then plug back in the Kinect on every boot. The auto-suspension does not appear to take effect until after the port is powered down the first time. Navigate to the bin directory where the samples reside:
cd libfreenect/build/bin
and run one of the sample programs:
sudo ./freenect-glview
Then you are ready to play and have some fun!
Permissions
If you do not want to run your programs using sudo, you can add rules to your udev rules directory. Place the file ’51-kinect.rules’ into /etc/udev/rules.d directory, and set the video permissions.
First change your directory to your libfreenect/platform/linux/udev directory, then:
$ sudo adduser $USER video
$ sudo cp 51-kinect.rules /etc/udev/rules.d/51-kinect.rules
Make sure to log out and then log back in.
For more information see: http://openkinect.org/wiki/Getting_Started#Ubuntu_Manual_Install
and: http://openkinect.org/wiki/Getting_Started#Use_as_normal_user
Note: Thanks to Corey Thompson for the pointing out the permissions issues. You can read more about this on the NVIDIA Jetson forum at: https://devtalk.nvidia.com/default/topic/751916/microsoft-kinect-with-jetson-tk1/?offset=33#4569415
12 Responses
I get an error:
Kinect camera testNumber of devices found: 1
Could not open audio: -4
Failed to open motor subddevice or it is not disabled.Failed to open audio subdevice or it is not disabled.Could not open device
Hi Alessandro
try to disable FREENECT_DEVICE_MOTOR, and only leave FREENECT_DEVICE_CAMERA. You have to do this in each place where kinect device is initialized, for instance in:
wrappers/c_sync/libfreenect_sync.c if you want to use c_sync wrapper
examples/glview.c
in particular in order to disable motor subsystem remove FREENECT_DEVICE_MOTOR:
freenect_select_subdevices(ctx, (freenect_device_flags)(FREENECT_DEVICE_MOTOR | FREENECT_DEVICE_CAMERA));
so the line becames:
freenect_select_subdevices(ctx, (freenect_device_flags)(FREENECT_DEVICE_CAMERA));
of course this is only a workaround, this not allow the use of the motor at least in 1473 kinect model
good luck!
-Alex
I have the same problem, but disabling motor subdevice as above didn’t work. Any other hint on how to fix it? Thanks!
I’m still in same problem.
Any other hint on how to fix it?
Hi Hisao,
I can’t replicate this issue because I don’t have the model of Kinect that is causing this problem. One thing to try is to install the ‘Grinch’ kernel, which now includes drivers for the Kinect. Instructions for installation can be found on the Jetson forum: https://devtalk.nvidia.com/default/topic/823132/embedded-systems/-customkernel-the-grinch-21-3-3-for-jetson-tk1-developed/
Sorry I can’t be of more help.
I also had some issues running the examples, even when lsusb was showing the Kinect. They were mostly permissions-related. I noted the steps I needed to take on the NVidia Developer discussion forum here: https://devtalk.nvidia.com/default/topic/751916/microsoft-kinect-with-jetson-tk1/?offset=33#4569415 of course, this is mostly just some pointers to make sure and follow all of the steps on http://openkinect.org/wiki/Getting_Started
Thanks Corey, those are certainly good tips! The Grinch kernel has Kinect Xbox 360 support built it, so it makes it a little bit easier. I’ve updated the article with the permissions information that you shared. The Kinect post and video were one of the first ones that I did, so revisiting with what I’ve learned since makes me wince a little. Cheers!
When I run the glview example the screen appears black and the light on the kinect becomes red
I forgot to mention that when I run the glview example I can control the motor and the led light on the kinect.
Hi,
I get an error:
[ 97%] Building CXX object wrappers/cpp/CMakeFiles/freenect-cpp_pcview.dir/cpp_pc_view.cpp.o
In file included from /home/ubuntu/libfreenect/wrappers/cpp/cpp_pc_view.cpp:30:0:
/home/ubuntu/libfreenect/wrappers/cpp/./libfreenect.hpp:182:3: error: ‘unique_ptr’ in namespace ‘std’ does not name a type
std::unique_ptr m_rgb_buffer;
^
/home/ubuntu/libfreenect/wrappers/cpp/./libfreenect.hpp: In member function ‘void Freenect::FreenectDevice::setVideoFormat(freenect_video_format, freenect_resolution)’:
/home/ubuntu/libfreenect/wrappers/cpp/./libfreenect.hpp:115:5: error: ‘m_rgb_buffer’ was not declared in this scope
m_rgb_buffer.reset(new uint8_t[mode.bytes]);
^
make[2]: *** [wrappers/cpp/CMakeFiles/freenect-cpp_pcview.dir/cpp_pc_view.cpp.o] Error 1
make[1]: *** [wrappers/cpp/CMakeFiles/freenect-cpp_pcview.dir/all] Error 2
make: *** [all] Error 2
during entering ‘make’.
Hey Amirul,
I am having the same problem too. My work around at the moment is to checkout d20b594.. right before the last commit. I already posted my problem to the repositories. I hope you can post something too saying that you are experiencing the same problem as I am.
Here is the issues i posted on OpenKinect/libfreenect repository:
https://github.com/OpenKinect/libfreenect/issues/576
Best Iman
sorry I meant `git checkout 3bf6576`.