Kinect on a Pioneer Mobile Robot with Onboard SBC

From MobileRobots Research and Academic Customer Support

Jump to: navigation, search

Contents

Kinect on a Pioneer Mobile Robot with Onboard SBC

Microsoft’s Kinect is an interesting 3D sensor potentially valuable to and certainly a source of interest for applications in mobile robotics, especially since it is available at such a very low price. It is based on Primesense Ltd’s unique IR-based structured light sensor and software that provides not only 3D ranging information, but can extract people and poses in its field of view, among many other uses. Images grabbed from its color camera, for example, also can be used for vision-based sensing.

Although initially targeted at Xbox 360 gamers, it was no wonder that soon after its introduction in mid-2010, people started applying it with a frenzy to many other PC-based applications. Now the open-source software is available free online and even Microsoft has gotten on the bandwagon with drivers and an SDK to support the rush of non-gamer enthusiasts. Many of MobileRobots’ platforms, including Pioneer 3 DXs and ATs, PeopleBots, PowerBots, Seekurs and Seekur Jr.s out there have a Versalogic single-board computer (SBC) onboard that can support the Kinect device. Any netbook or laptop with a USB port can also be used.

Pioneer robots support the power (12VDC) and signal (USB 2.0) requirements of the Kinect easily. Read below for notes on connecting the Kinect.

Several organizations and projects are working on Kinect software. These include but are certainly not limited to OpenNI, OpenKinect, ROS and Microsoft.

To integrate Kinect on Windows, simply use Microsoft’s Kinect for Windows SDK.

For Linux, there are several options, described below.

Power and Data Connections

Kinects sold separately from X-Box consoles come with an 110 or 220 V AC to 12 V DC wall wart transformer and USB plug, which are combined into the special X-Box aux socket (with angled corner and marked with orange) which the Kinect's connector attaches to. This power supply adapter part is also available for purchase separately here on the Microsoft online store.

If you have the new dual core Versalogic Mamba EBX-37 Onboard Computer, or a previous computer but your robot was ordered with the a SICK LMS-200 laser rangefinder accessory and laser navigation package, standard USB sockets are available on the exterior of the robot (and also on the computer board itself in the case of the Mamba). Otherwise, you’ll need to contact support to purchase a CAB1677 USB Connector for Cobra EBX-12 Onboard Computer.

Cut off the AC/DC transformer of the Kinect power cable (Note: you can purchase a spare Kinect power supply here on the Microsoft online store if you want to keep one unmodified for use off the robot). There is more than enough wire, so clip it at a reasonable length to reach from the Kinect device mounted on Pioneer, through the robot’s access port and into the Pioneer 3 Motor-Power Board. Strip about ½ centimeter (1/4 inch) and solder tin the white and brown power wires. White is GND (common) and the brown wire is +12V. (Don’t get them backwards!)

Attach the power wires to your robot’s Motor/Power distribution board’s AUX 12V (brown wire) and Ground (white wire) connectors. Find the details in the Pioneer 3 Operations Manual Appendix B for the location of those connectors.

See Connecting Devices to the Pioneer 3 Motor-Power Board for instructions on opening the robot and accessing the motor/power board.

NOTA BENE: 

Make power and hardware modifications to your robot WITH CAUTION AND AT YOUR OWN RISK. 
MobileRobots cannot be held responsible for any mistakes in this document nor any mistakes or failures that 
you may make in performing the procedures described herein.

Image:Accessory_Power_on_P3_MPB_12_screw_terminals_using_screwdriver_(Small).JPG‎

Mounting on the Robot

Finally, you can mount the Kinect on the robot.

Adhesive-backed Velcro pieces makes it easy to remove the Kinect during maintainence or other robot use.

For a more permanent solution, various third-party mount adapters are available for sale online that attach to the Kinect base and provide a standard photo/video 1/4"-20 screw hole usually seen on tripods etc. The Pioneer Camera Mounting Plate includes a 1/4 inch diameter hole in the center, recessed on the bottom to allow a flat screw head. The Kinect tripod mount can be screwed to this plate using a short (about 3/8"-1/2") 1/4"-20-thread machine screw with a flat head through this hole, and then the plate may be screwed onto the robot top plate using the M4 screws already in the top plate. See Pioneer Camera Mounting Plate for photographs and instructions. Contact Support to order this plate. This plate may be used with Pioneer 3 DX and AT, PeopleBot, PowerBot and Research PatrolBot robots.

OpenNI

This section describes how to get the OpenNI software working.

OpenNI currently provides open-source, free Kinect driver and library software in source and binary forms for 32-bit Windows XP or 7 , Ubuntu (10.10, Maverick Meerkat) Linux, and even OSX. The tricky part is that most Pioneers shipped with Linux—Debian 5 or earlier—means you’ll have to change your OS. (Windows Embedded XP works fine.) The trickier part is that the Kinect driver binaries are compiled with the SSE3 Pentium 4 processor code-based extensions enabled. The Cobra EBX-12 computer used on many robots only supports SSE2. The bad news is that Primesense’s NITE middleware for gesture recognition and control comes as binaries only, compiled with the SSE3 flag set, so NITE cannot be used on these computers—they crashe with a hardware-based Illegal instruction error.

There is good news, though. Pioneer SDK software, including ARIA and ARNL, work without a hitch on upgraded OSes. And the OpenNI driver and SDK software can be recompiled with SSE2 extensions only (SSE3 disabled) .


Changing the OS to Ubuntu

If you adopt Ubuntu, replace the hard disk drive altogether and save the old one as a backup. That way you’ll protect the original. Versalogic Cobra uses an IDE bus, not SATA. We replaced the drive with an 80 GB 2.5-inch one. Be careful to use a new or fully reformatted (low-level) drive. We ran into difficulties booting a disk into Ubuntu when it had previously been installed with Debian Linux. We installed the new OS from a downloaded DVD image. Access to the onboard PC is through the Pioneer’s nose. Attach and enable an externally powered DVD drive according to the instructions given at [1]. The procedures includes changing the SBC’s BIOS, so also attach a keyboard and monitor. Hit the Del key just after starting power to get to the BIOS editor, and be patient.

Ubuntu 10.10 Server

The Versalogic Cobra SBC’s display hardware does not support Ubuntu’s usual X display configuration. Accordingly, download the server, not the client version and boot it on the robot’s SBC-attached DVD. Choose mostly the default options during installation. Select to install OpenSSH if you want to access the SBC over the network. When finished installation, remove the CD option from the BIOS boot-order list in order to speed it up.

Ubuntu doesn’t have a root login at the outset. The username and password you entered during installation is on the sudo’ers list, however. So either perform much of the following instructions as that user with the ‘sudo’ preface to each command line, or enable root by setting its password and then re-login as root: sudo passwd root.

You’ll need to install a couple basic things to proceed. First, the compiler: apt-get install g++ (you may need sudo, too). And you’ll need some wireless tools to help get your robot’s wireless interface up and running: apt-get install wireless-tools. And you’ll need the USB libraries: apt-get install libusb-1.0.

Ubuntu Networking

You’ll need to set up the network. The wireless card’s drivers should have installed automatically during installation of the OS. If not, contact support. Edit /etc/network/interfaces. For example, add: 

 auto eth0
 iface eth0 inet dhcp

 auto eth1
 iface eth1 inet static
 address 192.168.100.100
 netmask 255.255.255.0
 network 192.168.100.0
 broadcast 192.168.100.255

 auto wlan0
 iface wlan0 inet static
 wireless-essid Wireless Network
 address 192.168.10.101
 netmask 255.255.255.0
 network 192.168.10.0
 broadcast 192.168.10.255

Use the wireless-key and/or wireless-mode options to enable security and/or select Ad-Hoc versus Managed (access point) connection mode. Use if-down and if-up with eth0, eth1 and/or wlan0 to bring the respective interface down or up. They all come up automatically if you use the auto specification in interfaces. Comment out an auto line to disable an Ethernet interface on start up.

Refer to the notes at http://robots.mobilerobots.com for further details on computer drivers and configuration, including network configuration.

OpenNI Drivers and SDK

The OpenNI drivers for the Kinect are open source. Because the EBX-12 “Cobra” onboard computer doesn’t support SSE3 (lots of other PCs don’t, either), you have to download the sources and compile your own.

Download the OpenNI driver software tree by visiting [2]. Click the “Download” button on the right side of the page to download the tar.gz for Linux or the .zip archive for Windows, or clone the branch using git. Read the README at the top-level of the OpenNI directory tree. Read especially for any requirements, such as Microsoft Visual Studio, Python v2.6 or later, and so on. Also be sure to install PyWin32 if using Windows—it provides some needed Python extensions. We also installed the optional packages— they made the recompilation easier.

Another thing to note is that the Kinect has its own drivers modified from the ones available from Primesense which work only with Primesense’s devices. Download the sources for these also from github: [3]; tar.gz for Linux or .zip for Windows.

Linux Installation

To compile the OpenNI libraries and samples under Ubuntu, navigate down the OpenNI directory tree to the Linux-x86 or Win32 directory under Platform. Enter into the CreateRedist directory and, from the current Linux terminal, type: export SSE_GENERATION=2, then execute RedistMaker: 

export SSE_GENERATION=2
cd Platform/Linux-x86/CreateRedist
./RedistMaker

Navigate up a directory to Redist. First, manually create a missing directory /var/lib/ni. Then run the install.sh script. Do these as root or use sudo: 

cd ../Redist
sudo mkdir /var/lib/ni
sudo ./install.sh

Similarly, to prepare and install the Kinect-specific drivers, navigate down the avin2 directory tree to Platform/Linux-x86/CreateRedist, and install the driver from the ../Redist/install.sh script. Voile’!

Find the demonstration programs in the Platform/Linux-x86/Redist/Samples/Bin/Release directory of OpenNI. NiViewer is the most versatile, showing live video with depth sensing. Very cool.

Unfortunately, the demos are using OpenGL for graphics display, which on the Cobra computer is not hardware accelerated, resulting in somewhat slow graphics display.

Windows Installation

Microsoft has now released a Kinect SDK. However, if you'd like to use OpenNI instead, here's how:

With Windows XP Embedded which comes with that option for the Pioneer Versalogic Cobra, the compilation didn’t work out of the box. Instead, we had to load the Platform\Win32\Build\OpenNI.sln’s from both the OpenNI and avin2 distributions into Microsoft Visual Studio (MVS). In doing so, all the solution projects get updated to the current version of MVS and .NET. Additionally, we cleaned the release and debug versions of the solutions (Build:Clean), before File:Save all and exiting.

As with the Linux version, start a cmd window and navigate down the respective driver directory trees to Platform\Win32\CreateRedist. Use “set SSE_GENERATION=2” to set that compile flag and then run the RedistMaker.bat script to create the OpenNI drivers. The installer gets put into the ..\Final directory. We used the “dev” version to get the samples installed, too.

Similarly, run the Redist.py script in avin2’s Platform\Win32\CreateRedist directory. Its device driver installer gets put into the CreateRedist\Output directory. Use the Control Panel:System:Hardware:Device Manager to verify that the Kinect Motor, Audio and Video device drivers are indeed installed (installation sequence happens when you first plug in the Kinect to the USB port, too). The samples NiViewer and NiSimpleRead get installed in the Start menu’s Programs:OpenNI option.

Resources

The wealth of source code for the OpenNI and avin2 drivers are in the respective top level Source directory. There is a very useful OpenNI User’s Manual in the Documentation directory, and the Samples contains the sources for the, yes, the samples.

OpenKinect

More about the OpenKinect project is at http://www.openkinect.org

The OpenKinect wiki also contains some techincal documentation on the Kinect internals, protocol, etc.

Using OpenCV to work with Kinect data

Thanks to Eleri Cardozo on pioneer-users for this information.

You can import video and depth map images from the OpenKinect library into OpenCV (a library for computer/machine vision and image analysis.)

For example, getting the image from OpenKinect: (the kinect_rgb_buffer object is from that library containing image data, while width and height specify the size of that image) 

// OpenKinect returns image in kniect_rgb_buffer in RGB forma, convert to OpenCV images in BGR:
IplImage* kFrame = cvCreateImage(cvSize(width, height), IPL_DEPTH_8U, 3); memcpy(kFrame->imageData, kinect_rgb_buffer, width*height*3);
IplImage* cFrame = cvCreateImage(cvGetSize(kFrame), IPL_DEPTH_8U, 3);
cvCvtColor(kFrame, cFrame, CV_RGB2BGR);   // RGB (kFrame) to BGR (cFrame)
cvReleaseImage(&kFrame); // destroy kFrame, only use cFrame from now on

Or getting the depth buffer from OpenKinect: (here kinect_depth_buffer contains depth data while width and height specify the size) 

// Images from depth camera have 2 bytes per pixel and is usually converted to // 1 byte per pixel for handling in OpenCV:
cv::Mat mydepth = cv::Mat(width, height, CV_16UC1, kinect_depth_buffer); cv::Mat depth8; mydepth.convertTo(depth8, CV_8UC1, 1.0/4.0);
IplImage* cFrame = cvCreateImage(cvSize(width, height), IPL_DEPTH_8U, 1); memcpy(cFrame->imageData, depth8.data, width*height);

A bit of technical detail and calibration info

(Thanks to Eleri Cardozo on pioneer-users for this pointer!)

A brief, useful discussion of the Kinect depth calculation and calibration is at http://www.ros.org/wiki/kinect_calibration/technical

ROS

ROS contains four Kinect drivers.

Retrieved from "http://robots.mobilerobots.com/wiki/Kinect_on_a_Pioneer_Mobile_Robot_with_Onboard_SBC"
Personal tools