Object Detection using blob tracing

Once, while I was working, I accidently spilled a whole bunch of small measuring cubes on the table. While I was cleaning the desk I was thinking ‘if only I had a pet robot to clean my desk…’. Yeah. I am that lazy. And it flashed the whole robotics course that I had taken in my gradschool. And I started thinking, how would the robot do this task? I ended up taking a few pictures of the same messy table and started processing those. Well, this is what I had.

Raw images:

What do I want to do with these images?

Assuming, this is the line of sight for the robot, let’s detect the dice in the images, of different colors, using OpenCV3 along with Python3 to try out the analysis.

How do I plan to do that?

Convert the RGB image into HSV image

The image we have is colored. Let’s convert it in the HSV Cylindrical coordinates. HSV - Hue, Saturation and Value - is a coordinate system which represents how our eyes perceive the colors. Hue: the type of chroma Saturation: the amount of white content in the chroma Value: the amount of blank content in the chroma

How could this be useful to us? Our image is a colored image. The objects we wish to detect are of different colors. So, why not use the information about the chroma to identify the objects.

On converting the image from RGB to HSV will give us the following result:

import numpy as np
import cv2

rawImage = cv2.imread('rawImage.jpg')
cv2.imshow('Original Image',rawImage)
cv2.waitKey(0)

hsv = cv2.cvtColor(rawImage, cv2.COLOR_BGR2HSV)
cv2.imshow('HSV Image',hsv)
cv2.waitKey(0)

Select Saturation

So, lets extract the colorfulness of the objects according to its brightness. The image extracted will give us good results for all the colors except black dice. That is because, the saturation information i.e. the white content for the black dice will be much less. To get the Saturation of the image, we will split the HSV image we obtained in the 1st step.

hue ,saturation ,value = cv2.split(hsv)
cv2.imshow('Saturation Image',saturation)
cv2.waitKey(0)

Threshold the image

The results on the previous step looks good. We can see some bright objects in the image. Now, let us convert them to binary. This makes it easier to extract the objects. To binarize the image, we will go for Otsu Thresholding. This will give us a thresholding value that best suits the image. It does this in the background: treats the image as a bimodal image, finds the histogram peaks that best describes the image and then, selects a value that lies between the two peaks.

retval, thresholded = cv2.threshold(s, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
cv2.imshow('Thresholded Image',thresholded)
cv2.waitKey(0)

Apply Median Filter

Our focus is to find the dice in the image. According to the image, thresholding often gives few areas with small patches. We don’t need these patches for our solution. Let’s get rid of them. Since it’s a binary image, we can easily use median filter which is best known for removing the noise in our image like the salt and pepper you can see near the corner. It calculates the median value in the neighborhood area and assigns that value to the pixel.

medianFiltered = cv2.medianBlur(thresholded,5)
cv2.imshow('Median Filtered Image',medianFiltered)
cv2.waitKey(0)

Detect Contours

At this point, we are done with the preprocessing steps. Now comes the part where we detect the objects. If we look at the image obtained in the previous step, we see that we have some large while sections in the black area. If we go back to the original image, we will also notice that these are the dice that we wanted to detect. So, let’s use a mathematical concept called contours. Contours are the shape that is formed by joining the points covering an area of similar intensity or color. In our image, we could think of it as the area that is formed by joining the boundary of white pixels. Contours work best when the image is binary. In OpenCV ‘findContours’ will locate white objects on a black background. This is one of several ways to find blobs(binary large objects) in the image using OpenCV.

_, contours, hierarchy = cv2.findContours(medianFiltered, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

Area of Contours

When we detect the contours, we still see some patches in the image that are unnecessary for detecting our object of interest. That is because the area of the object is way too small to give us any relevant information for our analysis. We can discard such objects by giving a condition to check if their area is below 100px. For example, in the images below look at the small red patches at the top. These are the objects detected. We are surely not looking to detect something like that.

contour_list = []
for contour in contours:
    area = cv2.contourArea(contour)
    if area > 100 :
        contour_list.append(contour)

Draw the detected Contours

We are in the final step. Now what do we need? Well, we want to see the objects that have been detected, so that the robot comes and picks it up.

cv2.drawContours(rawImage, contour_list,  -1, (255,0,0), 2)
cv2.imshow('Objects Detected',rawImage)
cv2.waitKey(0)

The complete source code for the algorithm is as follows:

import numpy as np
import cv2

rawImage = cv2.imread('rawImage.jpg')
cv2.imshow('Original Image',rawImage)
cv2.waitKey(0)

hsv = cv2.cvtColor(rawImage, cv2.COLOR_BGR2HSV)
cv2.imshow('HSV Image',hsv)
cv2.waitKey(0)

hue ,saturation ,value = cv2.split(hsv)
cv2.imshow('Saturation Image',saturation)
cv2.waitKey(0)

retval, thresholded = cv2.threshold(s, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
cv2.imshow('Thresholded Image',thresholded)
cv2.waitKey(0)

medianFiltered = cv2.medianBlur(thresholded,5)
cv2.imshow('Median Filtered Image',medianFiltered)
cv2.waitKey(0)

_, contours, hierarchy = cv2.findContours(medianFiltered, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

contour_list = []
for contour in contours:
    area = cv2.contourArea(contour)
    if area > 100 :
        contour_list.append(contour)

cv2.drawContours(rawImage, contour_list,  -1, (255,0,0), 2)
cv2.imshow('Objects Detected',rawImage)
cv2.waitKey(0)

So, looks like the objects detected aren’t too bad. Also, we have come up with an algorithm. That’s cool. This algorithm works well for colored objects. The brighter the objects, better is the recognition. In our example, we saw that the background contained neutral colors.

Well, now that we have learnt one of the several ways of detecting objects , let’s make my robot a bit more intelligent next time. Maybe it would detect objects of several shapes in a bit more cluttered environment. Till then have a great time.

I hope you found this useful. Please share any tips to make this better.