OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection

This is a recurring subject in Stackoverflow and since I was unable to find a relevant implementation I decided to accept the challenge.

I made some modifications to the squares demo present in OpenCV and the resulting C++ code below is able to detect a sheet of paper in the image:

void find_squares(Mat& image, vector<vector<Point> >& squares){    // blur will enhance edge detection    Mat blurred(image);    medianBlur(image, blurred, 9);    Mat gray0(blurred.size(), CV_8U), gray;    vector<vector<Point> > contours;    // find squares in every color plane of the image    for (int c = 0; c < 3; c++)    {        int ch[] = {c, 0};        mixChannels(&blurred, 1, &gray0, 1, ch, 1);        // try several threshold levels        const int threshold_level = 2;        for (int l = 0; l < threshold_level; l++)        {            // Use Canny instead of zero threshold level!            // Canny helps to catch squares with gradient shading            if (l == 0)            {                Canny(gray0, gray, 10, 20, 3); //                 // Dilate helps to remove potential holes between edge segments                dilate(gray, gray, Mat(), Point(-1,-1));            }            else            {                    gray = gray0 >= (l+1) * 255 / threshold_level;            }            // Find contours and store them in a list            findContours(gray, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);            // Test contours            vector<Point> approx;            for (size_t i = 0; i < contours.size(); i++)            {                    // approximate contour with accuracy proportional                    // to the contour perimeter                    approxPolyDP(Mat(contours[i]), approx, arcLength(Mat(contours[i]), true)*0.02, true);                    // Note: absolute value of an area is used because                    // area may be positive or negative - in accordance with the                    // contour orientation                    if (approx.size() == 4 &&                            fabs(contourArea(Mat(approx))) > 1000 &&                            isContourConvex(Mat(approx)))                    {                            double maxCosine = 0;                            for (int j = 2; j < 5; j++)                            {                                    double cosine = fabs(angle(approx[j%4], approx[j-2], approx[j-1]));                                    maxCosine = MAX(maxCosine, cosine);                            }                            if (maxCosine < 0.3)                                    squares.push_back(approx);                    }            }        }    }}

After this procedure is executed, the sheet of paper will be the largest square in vector<vector<Point> >:

I'm letting you write the function to find the largest square. ;)

Unless there is some other requirement not specified, I would simply convert your color image to grayscale and work with that only (no need to work on the 3 channels, the contrast present is too high already). Also, unless there is some specific problem regarding resizing, I would work with a downscaled version of your images, since they are relatively large and the size adds nothing to the problem being solved. Then, finally, your problem is solved with a median filter, some basic morphological tools, and statistics (mostly for the Otsu thresholding, which is already done for you).

Here is what I obtain with your sample image and some other image with a sheet of paper I found around:

The median filter is used to remove minor details from the, now grayscale, image. It will possibly remove thin lines inside the whitish paper, which is good because then you will end with tiny connected components which are easy to discard. After the median, apply a morphological gradient (simply dilation - erosion) and binarize the result by Otsu. The morphological gradient is a good method to keep strong edges, it should be used more. Then, since this gradient will increase the contour width, apply a morphological thinning. Now you can discard small components.

At this point, here is what we have with the right image above (before drawing the blue polygon), the left one is not shown because the only remaining component is the one describing the paper:

Given the examples, now the only issue left is distinguishing between components that look like rectangles and others that do not. This is a matter of determining a ratio between the area of the convex hull containing the shape and the area of its bounding box; the ratio 0.7 works fine for these examples. It might be the case that you also need to discard components that are inside the paper, but not in these examples by using this method (nevertheless, doing this step should be very easy especially because it can be done through OpenCV directly).

For reference, here is a sample code in Mathematica:

f = Import["http://thwartedglamour.files.wordpress.com/2010/06/my-coffee-table-1-sa.jpg"]f = ImageResize[f, ImageDimensions[f][[1]]/4]g = MedianFilter[ColorConvert[f, "Grayscale"], 2]h = DeleteSmallComponents[Thinning[     Binarize[ImageSubtract[Dilation[g, 1], Erosion[g, 1]]]]]convexvert = ComponentMeasurements[SelectComponents[     h, {"ConvexArea", "BoundingBoxArea"}, #1 / #2 > 0.7 &],      "ConvexVertices"][[All, 2]](* To visualize the blue polygons above: *)Show[f, Graphics[{EdgeForm[{Blue, Thick}], RGBColor[0, 0, 1, 0.5],      Polygon @@ convexvert}]]

If there are more varied situations where the paper's rectangle is not so well defined, or the approach confuses it with other shapes -- these situations could happen due to various reasons, but a common cause is bad image acquisition -- then try combining the pre-processing steps with the work described in the paper "Rectangle Detection based on a Windowed Hough Transform".

Well, I'm late.

In your image, the paper is white, while the background is colored. So, it's better to detect the paper is Saturation(饱和度) channel in HSV color space. Take refer to wiki HSL_and_HSV first. Then I'll copy most idea from my answer in this Detect Colored Segment in an image.

Main steps:

1. Read into BGR
2. Convert the image from bgr to hsv space
3. Threshold the S channel
4. Then find the max external contour(or do Canny, or HoughLines as you like, I choose findContours), approx to get the corners.

This is my result:

The Python code(Python 3.5 + OpenCV 3.3):

#!/usr/bin/python3# 2017.12.20 10:47:28 CST# 2017.12.20 11:29:30 CSTimport cv2import numpy as np##(1) read into  bgr-spaceimg = cv2.imread("test2.jpg")##(2) convert to hsv-space, then split the channelshsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)h,s,v = cv2.split(hsv)##(3) threshold the S channel using adaptive method(`THRESH_OTSU`) or fixed threshth, threshed = cv2.threshold(s, 50, 255, cv2.THRESH_BINARY_INV)##(4) find all the external contours on the threshed S#_, cnts, _ = cv2.findContours(threshed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)cnts = cv2.findContours(threshed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[-2]canvas  = img.copy()#cv2.drawContours(canvas, cnts, -1, (0,255,0), 1)## sort and choose the largest contourcnts = sorted(cnts, key = cv2.contourArea)cnt = cnts[-1]## approx the contour, so the get the corner pointsarclen = cv2.arcLength(cnt, True)approx = cv2.approxPolyDP(cnt, 0.02* arclen, True)cv2.drawContours(canvas, [cnt], -1, (255,0,0), 1, cv2.LINE_AA)cv2.drawContours(canvas, [approx], -1, (0, 0, 255), 1, cv2.LINE_AA)## Ok, you can see the result as tag(6)cv2.imwrite("detected.png", canvas)

Related answers:

1. How to detect colored patches in an image using OpenCV?
2. Edge detection on colored background using OpenCV
3. OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection
4. How to use `cv2.findContours` in different OpenCV versions?