AI Grand Prix — Vision & Detection Guide

1. Why Vision Matters

The entire perception-to-control loop is a three-step chain:

1. Detect gate in the camera frame (2D pixel coordinates)
2. Extract 4 corners at sub-pixel accuracy
3. PnP solve to recover distance and 6DoF pose relative to the gate

Corner accuracy directly determines depth accuracy. A 2px error at 10m range produces roughly 0.5m of depth error — enough to cause a collision or a missed gate. This is why we use U-Net segmentation with RANSAC corner extraction instead of simple bounding boxes.

2. Detection Mode Comparison

Three detection backends are available. Selection is controlled by VisionSettings.mode in race_config.py:

# race_config.py — VisionSettings
mode: str = "unet"    # "color" (VQ1), "yolo" (VQ2), or "unet" (primary)

3. U-Net Architecture (GateSegNet)

Overview

GateSegNet is a lightweight 4-level encoder-decoder with skip connections. It predicts a binary gate mask at full input resolution.

Architecture Diagram

ConvBlock Detail

Each ConvBlock consists of two sequential Conv3x3 + BatchNorm + ReLU layers:

class ConvBlock(nn.Module):
    # Conv2d(in_ch, out_ch, 3, padding=1)
    # BatchNorm2d(out_ch)
    # ReLU(inplace=True)
    # Conv2d(out_ch, out_ch, 3, padding=1)
    # BatchNorm2d(out_ch)
    # ReLU(inplace=True)

Encoder Path

Decoder Path

4. RANSAC Corner Extraction Algorithm

Step-by-Step Pipeline

Step 1 — Contour extraction. Threshold the U-Net sigmoid mask at 127 to produce a binary image, then run cv2.findContours(binary, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE). Filter by minimum area (200px) and aspect ratio (0.2 to 5.0).

Step 2 — Edge template. For each valid contour, compute the minimum-area rotated rectangle via cv2.minAreaRect. Extract 4 box corners with cv2.boxPoints, then order them as TL, TR, BR, BL. These define 4 template edge segments: TL→TR, TR→BR, BR→BL, BL→TL.

Step 3 — Point assignment. Each contour point is assigned to its nearest edge using point-to-line-segment distance. This partitions the contour into 4 clusters, one per gate edge.

for p in contour_points:
    best_edge = argmin([point_line_dist(p, edge_a, edge_b)
                        for edge_a, edge_b in edges])
    buckets[best_edge].append(p)

Step 4 — RANSAC line fitting (per edge cluster):

1. Sample 2 random points from the cluster → define a candidate line
2. Compute perpendicular distance from all cluster points to the candidate
3. Count inliers within threshold (3px)
4. Repeat for 50 iterations, keep the line with the most inliers
5. Refit on inliers using SVD: compute mean of inlier points, then principal direction via np.linalg.svd(inliers - mean)

# RANSAC core loop (simplified)
for _ in range(50):
    p1, p2 = random_sample(edge_pts, 2)
    direction = normalize(p2 - p1)
    normal = [-direction[1], direction[0]]
    dists = abs((pts - p1) @ normal)
    inliers = pts[dists < 3.0]
    if len(inliers) > best_count:
        mean = inliers.mean(axis=0)
        _, _, vt = np.linalg.svd(inliers - mean)
        best_line = (mean, vt[0])  # point + direction

Step 5 — Line intersection. Adjacent RANSAC-fitted lines are intersected analytically to produce 4 sub-pixel corner points. The intersection of lines L_i and L_i+1 gives corner C_{(i+1) mod 4}.

# Line intersection: p1 + t*d1 = p2 + s*d2
det = d1[0]*(-d2[1]) - d1[1]*(-d2[0])
if abs(det) < 1e-8: return None  # parallel
dp = p2 - p1
t = (-d2[1]*dp[0] + d2[0]*dp[1]) / det
corner = p1 + t * d1

Step 6 — cornerSubPix polish. OpenCV sub-pixel refinement on the mask image with a 5x5 search window, 30 iterations, epsilon 0.01:

criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.01)
cv2.cornerSubPix(gray_mask, corners, (5, 5), (-1, -1), criteria)

Step 7 — Corner ordering. Final corners are sorted into TL, TR, BR, BL order using coordinate sum and difference:

• TL = argmin(x + y)
• BR = argmax(x + y)
• TR = argmin(y - x)
• BL = argmax(y - x)

RANSAC Corner Extraction Diagram

5. PnP Depth Estimation

The Problem

Given 4 known 3D points (gate corners) and their 2D pixel positions (from RANSAC), recover the gate's 6DoF pose relative to the camera — specifically, the translation vector whose magnitude gives the distance.

Knowns

GATE_CORNERS_3D = [
  [-1.0,  1.0,  0],  # TL
  [ 1.0,  1.0,  0],  # TR
  [ 1.0, -1.0,  0],  # BR
  [-1.0, -1.0,  0],  # BL
]

Resolution: 640 x 480
FOV: 90deg H, 60deg V

fx = 640 / (2 * tan(45deg))
   = 320 / 1.0
   = 320.0

fy = 480 / (2 * tan(30deg))
   = 240 / 0.5774
   = 415.7

K = [[320.0,   0, 320],
     [  0, 415.7, 240],
     [  0,     0,   1]]

Solver

success, rvec, tvec = cv2.solvePnP(
    GATE_CORNERS_3D,     # (4,3) float64 — known 3D positions
    corners_2d,          # (4,2) float64 — detected pixel corners
    K,                   # 3x3 camera matrix
    dist_coeffs,         # distortion (zeros for now)
    flags=cv2.SOLVEPNP_IPPE_SQUARE  # optimized for coplanar squares
)

The SOLVEPNP_IPPE_SQUARE flag uses the Infinitesimal Plane-based Pose Estimation method, specifically optimized for coplanar square markers. It is both faster and more accurate than the general SOLVEPNP_ITERATIVE for this geometry.

Output

Fallback Distance Estimation

When PnP fails (degenerate corners, colinear points), a simple pinhole-model estimate is used:

distance = (GATE_WIDTH * fx) / pixel_width
         = (2.0 * 320.0) / bbox_w

6. Color Detection (VQ1 Mode)

In VQ1, gates are visually highlighted with a saturated color against a desaturated environment. Color detection is trivially fast (~0.5ms) and needs no training.

Pipeline

1. Convert BGR → HSV: cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
2. Threshold: cv2.inRange(hsv, hsv_lower, hsv_upper)
3. Morphological cleanup: close(5x5) then open(5x5) — fills small gaps, removes speckle
4. Contour extraction: cv2.findContours(mask, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE)
5. Filter: area > 500px, aspect ratio 0.3 to 3.0
6. Corner fitting: approxPolyDP at 4% arc length epsilon; falls back to bounding rect if not 4 vertices

Color Presets

# Switch preset at runtime
pipe.detector.set_color_preset("cyan")

7. YOLO Detection (VQ2 Mode)

For VQ2 environments where gates are not highlighted, a trained neural network detector is required. We use YOLOv8n or YOLOv11n for single-class gate detection.

Model Format Priority

The detector automatically resolves the best available format at load time (engine > onnx > pt).

Configuration

# race_config.py
yolo_model_path: str = "gate_detector.pt"
yolo_conf_threshold: float = 0.5

When to Use YOLO

• U-Net model is not yet trained
• VQ2 environment with complex backgrounds where color detection fails
• Quick prototyping (YOLOv8n trains in ~1 hour on a single GPU)

8. Training Workflows

U-Net Training

python gate_segmentation.py train --data dataset_gates_seg

Dataset structure:

dataset_gates_seg/
  train/
    images/    # BGR frames (PNG/JPG)
    masks/     # Binary masks (white = gate, black = background)
  val/
    images/
    masks/

Export to ONNX:

python gate_segmentation.py export --weights best.pt
# Produces: gate_seg.onnx (opset 17, dynamic batch axis)

YOLO Training

# Step 1: Auto-label using VQ1 color detection
python yolo-auto-label.py

# Step 2: Train
python yolo-train.py train

# Step 3: Export for deployment
python yolo-train.py export
# Produces: gate_detector.engine (TensorRT FP16) or gate_detector.onnx

9. Camera Calibration

Current Approach: Computed from FOV

Camera intrinsics are derived from the declared field-of-view angles:

fx = width  / (2 * tan(fov_h / 2))  # 640 / (2 * tan(45deg)) = 320.0
fy = height / (2 * tan(fov_v / 2))  # 480 / (2 * tan(30deg)) = 415.7
cx = width  / 2                       # 320.0
cy = height / 2                       # 240.0

# Distortion coefficients assumed zero
dist_coeffs = [0, 0, 0, 0, 0]

This is implemented in vision_pipeline.py as the CameraConfig class:

@property
def fx(self) -> float:
    return self.width / (2.0 * np.tan(np.radians(self.fov_h / 2)))

@property
def fy(self) -> float:
    return self.height / (2.0 * np.tan(np.radians(self.fov_v / 2)))

Better Approach: Checkerboard Calibration

For real hardware or high-fidelity simulation, a proper calibration produces a more accurate intrinsic matrix K and distortion coefficients:

1. Print a checkerboard pattern (e.g. 9x6 inner corners)
2. Capture 15-20 images from varied angles and distances
3. Run cv2.calibrateCamera() to solve for K and distortion
4. Update CameraConfig with the calibrated values

# OpenCV calibration (sketch)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
objp = np.zeros((6*9, 3), np.float32)
objp[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2) * square_size

obj_points, img_points = [], []
for img in calibration_images:
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    ret, corners = cv2.findChessboardCorners(gray, (9, 6), None)
    if ret:
        corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
        obj_points.append(objp)
        img_points.append(corners)

ret, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, gray.shape[::-1], None, None
)
print(f"Reprojection error: {ret:.4f} px")  # should be < 0.5

10. Pipeline Integration Summary

The complete vision pipeline is orchestrated by VisionPipeline in vision_pipeline.py:

from vision_pipeline import VisionPipeline, CameraConfig

cam = CameraConfig(width=640, height=480, fov_h=90.0, fov_v=60.0)
pipe = VisionPipeline(cam, mode="unet")  # or "color", "yolo"
pipe.setup()  # loads model weights

# Per frame:
detections = pipe.process(frame)       # detect + PnP in one call
nearest = pipe.get_nearest_gate()      # closest valid gate
latency = pipe.inference_latency_ms    # total ms for detect + PnP

Data Flow

Key Source Files