OpenCV: detect flawed rectangle

Question

currently I'm working on a project where I try to find the corners of the rectangle's surface in a photo using OpenCV (Python, Java or C++)

I've selected desired surface by filtering color, then I've got mask and passed it to the cv2.findContours:

cnts, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cnt = sorted(cnts, key = cv2.contourArea, reverse = True)[0]
peri = cv2.arcLength(cnt, True)
approx = cv2.approxPolyDP(cnt, 0.02*peri, True)
if len(approx) == 4:
    cv2.drawContours(mask, [approx], -1, (255, 0, 0), 2)

This gives me an inaccurate result:

Using cv2.HoughLines I've managed to get 4 straight lines that accurately describe the surface. Their intersections are exactly what I need:

edged = cv2.Canny(mask, 10, 200)
hLines = cv2.HoughLines(edged, 2, np.pi/180, 200)
lines = []
for rho,theta in hLines[0]:
    a = np.cos(theta)
    b = np.sin(theta)
    x0 = a*rho
    y0 = b*rho
    x1 = int(x0 + 1000*(-b))
    y1 = int(y0 + 1000*(a))
    x2 = int(x0 - 1000*(-b))
    y2 = int(y0 - 1000*(a))
    cv2.line(mask, (x1,y1), (x2,y2), (255, 0, 0), 2)
    lines.append([[x1,y1],[x2,y2]])

The question is: is it possible to somehow tweak findContours?

Another solution would be to find coordinates of intersections. Any clues for this approach are welcome :)

Can anybody give me a hint how to solve this problem?

Answer 1

Finding intersection is not so trivial problem as it seems to be, but before the intersection points will be found, following problems should be considered:

1. The most important thing is to choose the right parameters for the HoughLines function, since it can return from 0 to an infinite numbers of lines (we need 4 parallel)
2. Since we do not know in what order these lines go, they need to be compared with each other
3. Because of the perspective, parallel lines are no longer parallel, so each line will have a point of intersection with the others. A simple solution would be to filter the coordinates located outside the photo. But it may happen that an undesirable intersection will be within the photo.
4. The coordinates should be sorted. Depending on the task, it could be done in different ways.

cv2.HoughLines will return an array with the values of rho and theta for each line. Now the problem becomes a system of equations for all lines in pairs:

def intersections(edged):
  # Height and width of a photo with a contour obtained by Canny
  h, w = edged.shape

  hl = cv2.HoughLines(edged,2,np.pi/180,190)[0]
  # Number of lines. If n!=4, the parameters should be tuned
  n = hl.shape[0]    
        
  # Matrix with the values of cos(theta) and sin(theta) for each line
  T = np.zeros((n,2),dtype=np.float32)
  # Vector with values of rho
  R = np.zeros((n),dtype=np.float32)

  T[:,0] = np.cos(hl[:,1])
  T[:,1] = np.sin(hl[:,1])
  R = hl[:,0]

  # Number of combinations of all lines
  c = n*(n-1)/2
  # Matrix with the obtained intersections (x, y)
  XY = np.zeros((c,2))
  # Finding intersections between all lines
  for i in range(n):
    for j in range(i+1, n):
      XY[i+j-1, :] = np.linalg.inv(T[[i,j],:]).dot(R[[i,j]])

  # filtering out the coordinates outside the photo
  XY = XY[(XY[:,0] > 0) & (XY[:,0] <= w) & (XY[:,1] > 0) & (XY[:,1] <= h)]
  # XY = order_points(XY) # obtained points should be sorted
  return XY

here is the result: