Back in 2001 Viola and Jones published a paper on a real time face detection framework capable of rapid image processing and high detection rates, one of the seminal works in computer vision. There were three main contributions. First is an image representation called the “Integral Image” which allows features to be computed quickly. The second is a simple and efficient classifier to select a small number of critical visual features from a large set of potential features. The third contribution is a method for combining classifiers in a “cascade” which allows back-ground regions of the image to be quickly discarded which allows more computation on promising face-like regions. Here’s the demo on the Jetson:
A complete description of the algorithm is a little beyond what I can cover here. If you’re interested you can read one of their papers here: Robust Real-Time Face Detection.
Note here that I haven’t done optimization of any sort, it’s just basically “thrown together” in the simplest manner. You’ll notice that the frame rate is only around 18fps when the detector is not running, around 12 when it is. If we got serious and built this for reals I’m sure we could do much better. However, for a quick demo of me making funny faces and wearing sun glasses and a baseball cap, I think it’s a good tradeoff! As an aside, when Viola and Jones originally implemented this on a PC in 2001, they operated on a 384×284 image at 15fps on an Intel Pentium III 700MHZ PC. Oh, how times change, and in a good way this time.
Another part of the algorithm is that there is a “training mode”, where a set of face and non-face training images are used. The face training set typically consists of several thousand hand labeled faces scaled and aligned, and a larger set of non-face images. For the demo, a relatively small 1MB of this distilled type of training information is used. For hard core usage of face detection several hundred megabytes of this information may be employed.
Fortunately for mankind, this is the enabling technology which allows everyone to wear mustaches, beards, nose glasses, and all sorts of silly things on their webcams. I guess it might have other applications, but I can’t think of any right off the bat.
The demo itself is a modified version of an example from Kyle McDonald’s openFrameworks add-on ofxCV package. ofxCV acts as a wrapper to interface with the NVIDIA accelerated OpenCV implementation on the Jetson TK1. OpenFrameworks has built in Gstreamer support which I used to grab the video from a Logitech c920 webcam.
10 Responses
Hello,
thank you for this wonderful demonstration,
I trying to open an example from Kyle McDonald’s openFrameworks with CodeBlocks.
but it failed with can’t find manythings… I tried to add lib to the project, but finally failed in something about opnGL@@
Do I missing something about the OfxCV? @@
I found that the installation of OpenFrameworks on the Jetson was a little bit of a black art.
I have a branch on Github of openFrameworks of for version 8.4 for the Jetson: https://github.com/jetsonhacks/openFrameworks/tree/jetson, which fixes the OpenGL/ES issues, along with some of the other quirks. The biggest hurdle is that most of the other embedded ARM systems use OpenGL ES, the Jetson seems happier with OpenGL. Make sure that you run the Ubuntu install_dependencies script in openFrameworks/scripts/linux/ubuntu to set everything up. One of the outstanding issues is that the tesselator doesn’t work correctly which can lead to all sorts of issues with UI components.
Hopefully this will help you get openFrameworks up and running. I haven’t been coding with it recently, but it should at least get you up and running.
Thank you kangalow 🙂
It seems that still had a lot to work out..
it’s totally a black art for me so far.
after run the dependencies, my OpenCV4Tegra is removed.
then -lopencv_* seems not working anymore…
I think maybe I should reflash my jetson?@@
then I run the compileOF.sh which feels good to see something about gstreamer, but finally shows a line after “Done!”.
chown: cannot access ‘../lib/linux/*’: No such file or directory
I think I didn’t set the project correctly (can’t even find ofMain.h) :'(
infact, I just want to grab a 1920X1080 resolution frame from c920 which might need h264, and do the facedetection with GPU@@ that’s why I want to use openframework…
or does it possible to use gstreamer directly in C++ file?
Thanks again, I will keep trying ><
It took me more than a few tries to get openFrameworks running on a Jetson, it’s not surprising that you would run into issues. The openFrameworks install overwrites a lot of the Tegra files and replaces them with the more generic ones. The easiest way to get things back to normal is to either reflash the Jetson or install OpenCV4Tegra. If you just reinstall OpenCV4Tegra, then you may run into issues with library conflicts and header include files. Reflash to get back to the baseline.
The good thing about openFrameworks is that all the source is included. You should be able to grep for gstreamer in the source code and you should be able to find the outline on how they implemented their calls to gstreamer, and how they go about decoding the pipeline. In general, you will need to write a pipeline in Gstreamer (something similar to https://gist.github.com/jetsonhacks/3e4bcbd8212cff1091dc ) and call this with Gstreamer, which will give you the buffer to pass to OpenCV for face detection. In the demo, I showed a Haar Cascade Classifier doing the face detection.
All of this is not straightforward, but can be done in regular C++ without openFrameworks, or in OpenCV itself if the Gstreamer camera wrapper options are compiled in. Unfortunately this something that requires much more than a simple blog entry or comment can explain to get it to work, but with a little perseverance you should be able to get it to run. Good luck!
My experience was that this is a non trivial project.
Thanks again Kangalow 🙂
It’s excited to see the gst-lauch working and show the 1080p image smoothly from c920
but programming … I’ve tried to get frame from gst-buffer in my main.cpp and find many things on the internet, however, I felt really frustrated
that I got no idea how to solve it… haha :'(
a lot of this kind of msgs …undefined reference to `gst_object_get_type’
I got to try it more times
Thanks
Best regards.
You’re welcome, and good luck!
Dear kangalow,
I am very interested in openframeworks on Jetson and your demo was amazing and I try to do it, but I get an error when compiling.
I have download a branch on Github of openFrameworks of for version 8.4 for the Jetson: https://github.com/jetsonhacks/openFrameworks/tree/jetson then I run the the compileOF.sh but I get an error :
In file included from ../../../libs/openFrameworks/app/ofAppEGLWindow.cpp:26:0:
../../../libs/openFrameworks/app/ofAppEGLWindow.h:59:21: fatal error: EGL/egl.h: No such file or directory
#include
^
compilation terminated.
make[1]: *** [../../../libs/openFrameworksCompiled/lib/linuxarmv7l/obj/Debug/libs/openFrameworks/app/ofAppEGLWindow.o] Error 1
make: *** [Debug] Error 2
there has been a problem compiling Debug OF library
please report this problem in the forums
chown: cannot access ‘../lib/linux/*’: No such file or directory
Can you provides basic guide in configuring openframeworks to be used on Jetson
Thanks
EGL does not appear to be installed on your machine. Try:
$ sudo apt-get install build-essential
Thank you for the video
my project is about emotion recognition and i wonder if the jetson kit can ensure my work and which platform and language would I work with?
Hi,
That sounds like quite an ambitious project.
Unfortunately there are so many questions to answer, I’m not sure I’m able to help much.
First, you’ll have to answer for yourself which programming language(s) you’re most comfortable with programming. I can say that the Jetson TK1 is a standard Ubuntu desktop, so it will run C, C++, Python, Java, etc. OpenCV, which a lot of people use for computer vision processing, has bindings for most of the major languages, including C++ and Python. I don’t think the challenge is the programming language in your case.
The second question that you’ll have to answer for yourself is what type of performance you need for your project. The Jetson is fast for an embedded processor (think of any of the current tablets or smart phones), but compared to a desktop PC with a NVIDIA GPU card, still quite slow.
The third question that you’ll have to answer for yourself is how you are going to do the emotion recognition. A lot of researchers are working on emotion recognition using Deep Learning which requires a large set of data to train the neural net. Basically you would have a data set of faces which are categorized into categories (angry, happy, sad, etc) which is used to train the neural net. In general, these data sets are pretty large (which requires a large amount of memory), and training takes a significant amount of time, even on desktop machines. Once the neural net is trained, it can be deployed on an embedded processor like the Tegra K1 on the Jetson.
Some researchers do the training in the cloud (like Amazon AWS), and then download trained net to run locally. This page is typical of that approach:
https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data
There are other ways to do the recognition, I’m sure you know more about the subject than I do.
Good luck on your project!