Lab 3: Ball Detection and Collection

CSCE 439/839: Robotics
Lab 3: Ball Detection and Collection

Instructor: Carrick Detweiler
carrick _at_ cse.unl.edu
University of Nebraska-Lincoln
Fall 2013

Started: Weds, October 23, 2013
Lab Checkpoint: Thurs, October 31, 2013
Lab Due Date: Thurs, November 14, 2013

1 Overview

In this lab you will locate a ball and then push the ball to a location defined by a visual landmark. This will requires that you are able to visually localize a ball, drive to it, then bring it to a location marked by a visual landmark. I am giving you the ball and landmark detection code, but you will have to adapt it, characterize it, and integrate it into your system.

Make sure to read through the entire lab before starting to implement these. You are expected to complete up to and including Section 3 for the checkpoint (mainly demonstrating both visual systems working); however, I would encourage you to have completed more than this by the time of the checkpoint.

Remember and review the safety instructions from Lab 1.

2 Visual Landmarks (20pts.)

On the course website there is link to download ROS modules needed to identify barcode-like visual landmarks (called landmarkSelfSim). Download this code and place it in your ros directory. It should compile with some warnings, but no errors. I have also provided a number of these landmarks for use in class. These are "self-similar landmarks"¹, which means they are easily and quickly identified by only looking along one scan line in an image, no matter how far away they are. In addition, they have a binary bar code on the side that uniquely identifies each landmark.

There are two different launch files for the vision code landmarkSelfSim.launch and

displayDebugImages.launch. The first launches the landmark detection system (along with the camera drivers) and the second is used to display the output images, which is useful for debugging.

Question: What message does the landmark detection code publish about the location of landmarks? What are the various fields and what do they mean?

Question: At what framerate do you detect landmarks on your computer? Does it matter how many landmarks are in the image? Test this on at least two computers and report the results. Make sure to describe the systems you tested this on (e.g. virtual machine, native, etc).

Question: What is the maximum range you reliably can detect that a landmark is in the image?

Question: What is the maximum distance that you can accurately identify the id of the landmarks?

Question: For the previous question, what impact does the angle of the landmark have on the identification?

Question: How well does the landmark detection code work if the landmark is partially covered or out of the frame?

Question: Do experiments to characterize and calibrate the height of the landmark to the distance away it is in the image. Derive an equation that fits your data so that you can get a distance estimate to the landmark when you detect it. Characterize the performance of the range estimation.

Question: 839 Only: Adapt the landmark detection code so that it can detect landmarks that are upside down. Describe how you did this (hint: you should only need to modify landmarkDetector.cpp. You may also want to publish an additional debug image showing the results). Make sure that you publish the locations of any upside down landmarks and that you can use the same visual servoing code on upside down landmarks as regular ones.

3 Ball Detection (15 pts.)

In the code download for this lab there is code to detect balls (called ballDetector). In the launch directory, there are two launch files (similar to the landmark detection code). The first, ballDetector.launch, launches the camera and ball detector code. The second, displayDebugImages.launch, will display three different images you can use for debugging. If you find that the ball detector slows down your system, you can disable the debug images and information by commenting out the #define BALLDETECTOR_DEBUG line.

The ball detector works by first converting the image to HSV color space. It then performs thresholding to filter out all pixels that are between a low and high HSV threshold. These values can be configured in the launch file (see ballDetector.launch for an example) and can be dynamically changed by running the command rosrun ballDetector configGUI.py. It then searches for the largest group of connected pixels that have similar height and width (as a crude approximation to find circular groups of pixels). The largest group that meets this criteria is selected as the "ball" in the image and a message is sent out with this ball location.

The debug images display the HSV image, thresholded image (white pixels indicate those that are between the low and high thresholds), and a marked up image with white pixels indicating the boundaries of connected groups. In addition, in this image the final ball location is marked with a circle. Use these images and the configGUI to pick good HSV low and high thresholds.

Question: How did you go about picking the HSV thresholds for various balls? Do this for a couple of different color balls and report on the results. How does lighting affect the values?

Question: How many locations per second does the ball detector detect? Does it change based on different HSV thresholds? Make sure to describe the systems you tested this on (e.g. virtual machine, native, etc).

Question: Do experiments to characterize and calibrate range to the ball based on the reported radius. Derive an equation that fits your data so that you can get a distance estimate to the ball when you detect it. Characterize the performance of the range estimation.

Question: 839 only: Write a new node that subscribes to the HSV debug image and automatically tunes the HSV values when a ball is placed on a white background. Describe the details of this node and how well it works compared to manual tuning.

4 Visual Servo (15pts.)

In the previous lab, you implemented a reactive controller that maintained a fixed distance from an object based on the IR sensor readings. In this lab, you will implement a similar behavior by visually servoing to center and maintain a fixed distance from a landmark. Using the landmark detection code implement a visual servo behavior. The visual servoing code should be able to follow a landmark with a particular ID at a particular distance (you should be able to easily change these parameters).

Question: Describe your visual servo implementation.

Question: How well can you perform visual servoing? If you start from far away, how quickly can you get to the target distance? Include useful plots to help characterize the performance (e.g. estimated distance over time). The characterization of the servoing behavior should be detailed.

Question: What do you do if you lose sight of the target?

5 Vision Arbitrator (5pts.)

Create a new launch file that will launch both the ball detector and landmark detector code.

Question: With both running, what detection rates do you achieve? Do they differ from when you only had one running?

Most of the time you will not need to detect landmarks at the same time as you will detect balls. In this section you should create a system that will allow you to switch between using either landmark, ball detection, or both. You should do this by creating a vision arbitrator that will only pass images on to these nodes when needed.

Question: Describe the structure and how you implemented your vision arbitrator.

6 Ball Collection (20pts.)

N.B. As is frequently the case, this section is the shortest length, but will likely take the most time. So plan accordingly!

Your goal in this section is to locate a ball, "collect it," and then bring it to a specified landmark number. The way you do this is up to you. I would recommend keeping it simple by perhaps adding a simple fork to the front of the hovercraft to easily collect the ball and push it along. You could also make use of the range finders to tell if the ball is in the bin so that you don't need to aim the camera down too low. Note that there may be multiple landmarks, but you get to choose the ball, although I will place it.

839 only: Your course will have 2 balls (of your choice), but you will have two destination landmarks (out of many) that you will have to bring one ball each to. I would recommend using a simple state machine to make it work (e.g. focus on a single ball at a time) and don't overly complicate it.

Question: Describe your approach and provide a detailed characterization of the performance. This is the only question for this section, but I expect a detailed explanation in this section.

7 To Hand In

You should designate one person from your group as the point person for this lab (each person needs to do this at least once over the semester). This person is responsible for organizing and handing in the report, but everyone must contribute to writing the text. You should list all group members and indicate who was the point person on this lab. Your lab should be submitted by handin, http://cse.unl.edu/~cse439/handin/, before the start of class on the due date. Include a pdf of your report and your source code in handin. Also make sure to bring in a printed copy to class on the due date.

Your lab report should have an introduction and conclusion and address the various questions (highlighted as Question: ) throughout the lab in detail. It should be well written and have a logical flow. Including pictures, charts, and graphs may be useful in explaining the results. There is no set page limit, but you should make sure to answer questions in detail and explain how you arrived at your decisions. You are also welcome to add additional insights and material to the lab beyond answering the required questions. The clarity, organization, grammar, and completeness of the report is worth 10 points of your lab report grade.

In addition to your lab report, you will demonstrate your system and what you accomplished up to this point to the instructor at the beginning of lab on the checkpoint date and due date. This is worth 15 points of your overall lab grade. You do not need to prepare a formal presentation, however, you should plan to discuss and demonstrate what you learned and accomplished in all sections of the lab. This presentation should take around 10 minutes.

Question: Please include your code with the lab report. Note that you will receive deductions if your code is not reasonably well commented. You should comment the code as you write it, do not leave writing comments until the end.

Question: Include an rqt_plot of your final system and comment on your overall system architecture.

Question: For everyone in your group how many hours did each person spend on this part and the lab in total? Did you divide the work, if so how? Work on everything together? Give details on the hours each group member worked (e.g. keep a list of how many hours per day each person worked and on what).

Question: Please discuss and highlight any areas of this lab that you found unclear or difficult.

Footnotes:

¹D. Scharstein and A. Briggs. Real-time recognition of self-similar landmarks. Image and Vision Computing, 19(11):763-772, September 2001.

File translated from T_EX by T_TH, version 3.89.
On 23 Oct 2013, 13:48.