The Stereolabs ZED Camera is introducing SDK 2.0 which includes support for the Jetson TX1 and Jetson TX2. Looky here:
Background
The ZED Camera is a stereo depth sensor which contains two 4 megapixel imagers. For each frame, a composite of the left and right images (side by side) are sent over USB 3.0 to a host computer. The host computer then constructs a depth image using a GPU.
The small package and relatively light weight make the ZED an interesting choice for robotics and other mobile applications.
There are a variety of other APIs in the ZED SDK which do further processing. These functions include such interesting tasks as 6-axis Positional Tracking (visual odometery, no markers needed) and large-scale 3D environment mapping.
Stereolabs has a Github repository with a lot of code for project support, which is worth checking out.
The Jetson acts as the host computer for the ZED in this article.
Installation
Stereolabs recently updated their website to facilitate better interaction. Currently, you can get the SDK here.
Installing the SDK is straightforward. Download it from the website, set the permissions, and then run the installer from a terminal. The installer gives the option of also installing sample code, useful for development purposes.
Installation Issues
While downloading and installing the SDK was simple, actually getting the ZED to work was quite challenging in the development environment here.
First, we started using the ZED camera here over a year ago. Before most of the tools could work, a firmware upgrade was needed for the camera.
The firmware must be flashed from a PC. The PC here is an older, slower machine and doesn’t appear to care much for all that is new and shiny. It runs Windows 8. The Stereolabs package was downloaded from their website and installed on the PC. The ZED was connected to the PC, and then the dreaded ‘MSVCP120.dll is missing‘ message popped up.
This is odd, as the installer made it a point to ask for confirmation when installing the related Microsoft Visual C++ Redistributable Package. Having had to support this very issue over the last decade or so, I knew how to fix it. As a developer, I understood. As a consumer, I was annoyed.
Note: In the reorganization of the website, Stereolabs now has a page for troubleshooting this very issue. Unfortunately it wasn’t around when I first encountered the issue.
The second installation issue is the ZED Camera calibration. The procedure itself is simple enough. Unfortunately the calibration is effected by the lighting conditions when the test is being run. So much so that about the only time a good calibration could be made is when the room was dark. In my house, the only time it’s dark is at night. So calibrating at night was the only option.
Worse, there are times after running what appears to be a successful calibration that the tools cannot ‘find’ the configuration files. My guess is that it actually found the calibration file, but the file contained a bad range of values and is rejected. In either case, as a developer I understand. As a consumer and user, not so much.
And to be clear, I did a fresh install of the OS, downloaded the ZED SDK, and tried to calibrate the camera a dozen times. I tried restoring calibrations that I knew worked previously, and they were not recognized either.
Performance
Once the SDK is installed and the ZED is calibrated, there are several tools and examples to explore. The ZED Depth Viewer provides a depth map and point cloud viewer, and a status on the number of frames that are being processed per second (Hz).
The frame rates for the Jetson TX2 and Jetson TX1:
| Jetson TX1 vs Jetson TX2 – ZED Depth Viewer 720p @ 60 fps – bigger is better | |
|---|---|
| Machine | Average Hz |
| Jetson TX1 (Maximum Clocks) | 30 |
| Jetson TX2 (Mode Max-Q) | 33 |
| Jetson TX2 (Mode Max-P) | 43 |
As we can see from the table, in low power mode the Jetson TX2 is about 10% faster than the Jetson TX1 in maximum performance mode. In maximum performance mode the TX2 is almost 50% faster. Remember that you must factor in that the input stream is 60 fps. Interesting.
Conclusion
So there you have it, our first look on the TX2 with the ZED API 2.0. We will be using this camera on the current iteration of the Jetson RACECAR, so we should be able to see how it works in an actual application.
12 Responses
Hey Jim, quick question, did you tried using the zed-ros-wrapper for the camera? I am trying to compile it but so far no luck. I guess it could be that I am trying to use the 2.0 version.
I haven’t tried it out yet, so I don’t have much to offer right now.
I’m did try the ROS wrapper and it can’t compile, despite I have OpenCV installed on my TX1, it can’t find the OpenCV.
If you can find a way to fix it, please share your solution
The Github repository for the Stereolabs ZED ROS Wrapper is here: https://github.com/stereolabs/zed-ros-wrapper
You can ask the developers of the ZED wrapper your questions. Thanks for reading!
while I was compiling race-car project on TX2, I encountered this error:
/home/nvidia/catkin_ws/src/zed_wrapper/src/zed_wrapper_nodelet.cpp:66:26: fatal error: zed/Camera.hpp: No such file or directory
compilation terminated.
zed_wrapper/CMakeFiles/ZEDWrapper.dir/build.make:62: recipe for target ‘zed_wrapper/CMakeFiles/ZEDWrapper.dir/src/zed_wrapper_nodelet.cpp.o’ failed
make[2]: *** [zed_wrapper/CMakeFiles/ZEDWrapper.dir/src/zed_wrapper_nodelet.cpp.o] Error 1
CMakeFiles/Makefile2:3965: recipe for target ‘zed_wrapper/CMakeFiles/ZEDWrapper.dir/all’ failed
make[1]: *** [zed_wrapper/CMakeFiles/ZEDWrapper.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs….
Could you recognize what cause this problem and how can I deal with that?
Thanks
I resolve the problem by download the newest zed_wrapper plugin from:
https://www.stereolabs.com/developers/release/2.0/#sdkdownloads_anchor
and replace the folder in catkin_ws/src.
It turned out the zed_wrapper file on the mit-racecar github was out-dated.
I’m glad you got it to work!
Has anyone done this on a carrier board instead of the development board. I have a Zed working on the dev board but the Zed SDK fails to connect on the Orbitty carrier board. I’m running the same Jetpack on both and same ZED SDK the only difference is some kernel mods from connecttech to get the Orbitty USB working
Hi, this is very interesting. I am wondering if the heavy calculations of stereo are all done on the processor in ZED? or are they done on the host machine such as Jetson Tx2?
The calculations are done onboard the Jetson. The ZED basically streams video, with each frame consisting of an image from each camera sensor side by side. Thanks for reading!
I have a TX2 flashed with custom image from Jetpack 3.3. It installed CUDA 9.0 where the ZED SDK requires CUDA 8.0. Is there a way to have multiple versions of CUDA on a TX2? I saw on the NVIDIA forums that the install has to use Jetpack for CUDA
Stereolabs has a ZED camera download for CUDA 9.0 on the Jetson. Please see: https://www.stereolabs.com/developers/release/2.5/