Depth and 3D
How two images become a distance: stereo, disparity, point clouds, and when a camera can replace a laser scanner.
The idea
Hold a finger in front of your nose and close each eye in turn. The finger jumps a long way, an object across the room barely at all. That jump is called disparity, and it is inversely proportional to distance.
Two cameras a known distance apart do exactly that systematically.
What it is good for
- Determining the volume of a pallet.
- Letting robot arms grasp.
- Checking whether a part is seated correctly.
- Measuring rooms without a tape measure.
Calibration and depth map
import cv2, numpy as np
# 1. Calibration with a chessboard - once per camera rig.
# At least 20 shots from various angles and distances.
pattern = (9, 6)
objp = np.zeros((pattern[0] * pattern[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:pattern[0], 0:pattern[1]].T.reshape(-1, 2) * 25.0 # 25 mm
# 2. Depth map from a calibrated stereo pair.
left = cv2.imread("left.png", cv2.IMREAD_GRAYSCALE)
right = cv2.imread("right.png", cv2.IMREAD_GRAYSCALE)
stereo = cv2.StereoSGBM_create(
minDisparity=0, numDisparities=128, # must be divisible by 16
blockSize=5, P1=8 * 5 ** 2, P2=32 * 5 ** 2,
uniquenessRatio=10, speckleWindowSize=100, speckleRange=2)
# SGBM returns disparity in 1/16 pixel as int16.
disp = stereo.compute(left, right).astype(np.float32) / 16.0
f, B = 700.0, 120.0 # focal length px, baseline mm
with np.errstate(divide="ignore"):
depth_mm = np.where(disp > 0, f * B / disp, 0)- Calibration is valid for this rig only. Moving a camera invalidates it.
- Both cameras must trigger in sync; otherwise moving scenes are systematically wrong.
- The number of disparity levels sets the measurable near range and the compute time.
- Results without valid disparity are zero, not infinity. Those points must be masked.
The basic equation
Worked through
With f = 700 pixels, B = 120 mm and Δd = 0.5 pixels:
| Distance Z | Disparity d | Depth error ΔZ | relative |
|---|---|---|---|
| 1 m | 84.0 px | 6 mm | 0.6 % |
| 2 m | 42.0 px | 24 mm | 1.2 % |
| 5 m | 16.8 px | 149 mm | 3.0 % |
| 10 m | 8.4 px | 595 mm | 6.0 % |
| 20 m | 4.2 px | 2,381 mm | 11.9 % |
Which gives the design rule: to measure to ten centimetres at ten metres you need
a B of around 700 mm or a much longer focal length. A stereo camera with a
12 cm baseline is unsuitable for measurement beyond about five metres, whatever
the algorithm.
The alternatives
| Method | Range | Accuracy | Weakness |
|---|---|---|---|
| Passive stereo | 0.3 to 10 m | falls quadratically | Textureless surfaces |
| Active stereo with pattern | 0.2 to 5 m | high | Sunlight |
| Time of flight | 0.5 to 8 m | constant, medium | Reflective surfaces |
| LiDAR | 1 to 200 m | high, constant | Price, low point density |
| Learned monocular | any | relative only | No absolute scale |
Monocular depth estimation
A network can produce a plausible depth map from a single image because it has learned how large objects typically are. The result is correct up to an unknown scale and offset:
For measurement this is usable only if at least one object of known size is in
frame to determine s and t. Without that anchor, statements in metres cannot
be substantiated.