object detection

Object detection answers two questions at once that image classification answers only one of: not just what is in the image, but where it is and how many instances there are. A detector has to propose candidate locations, decide what's in each one, and then throw away the duplicates — three sub-problems that every major architecture since 2014 has solved in a different order, and that ordering is the real axis the field's history runs along.

R-CNN and its descendants solved detection by making "propose, then classify" fast enough to use. Girshick and colleagues' original R-CNN (2014) ran a separate region-proposal algorithm to generate around two thousand candidate boxes per image, then ran a full CNN forward pass on each cropped region independently — accurate, but slow enough that it processed well under one image per second. Fast R-CNN (2015) fixed the redundant computation by running the CNN once over the whole image and pooling features for each proposed region from that single feature map instead of recomputing from scratch. Faster R-CNN (Ren and colleagues, 2015) went further and replaced the external proposal algorithm with a Region Proposal Network — a small neural network that learns to propose regions directly from the same shared features, making the entire pipeline trainable end to end. This two-stage structure — propose candidate regions, then classify and refine each one — remains the accuracy-oriented baseline the field still benchmarks against.

YOLO made detection fast by giving up the separate proposal stage entirely. Redmon and colleagues (2016) reframed detection as one regression problem: divide the image into a grid, and have each grid cell directly predict a fixed number of bounding boxes, their confidence, and their class, all in a single forward pass through one network — "you only look once," as opposed to the two-stage approach's two passes. The tradeoff at the time was accuracy on small and crowded objects, since each grid cell could only commit to predicting a limited number of boxes, but the speed made real-time detection practical on video for the first time. SSD (Liu and colleagues, 2016) kept the one-stage, single-pass idea but predicted from multiple feature map resolutions simultaneously rather than one grid, which improved detection across a wider range of object sizes. The YOLO lineage has continued iterating ever since and remains the default choice whenever detection has to run in real time on constrained hardware.

Non-maximum suppression is the unglamorous step that turns a flood of candidate boxes into a clean result. A detector — one-stage or two-stage — typically proposes far more boxes around any real object than the one correct answer, since nearby grid cells or proposals all fire on the same object with slightly different positions. NMS sorts candidates by confidence, keeps the highest-scoring box, and discards every other box that overlaps it above an IoU threshold, repeating until no candidates remain; what's left is one box per object instead of a cluster. It's a simple greedy algorithm, but it's also a sharp edge — two genuinely separate, overlapping objects (a crowd of people, a stack of boxes) can have one correctly get suppressed as if it were a duplicate of the other, which is one of the harder failure modes for any NMS-based detector to avoid.

Focal loss fixed the class imbalance that was holding one-stage detectors back from matching two-stage accuracy. Lin and colleagues (RetinaNet, 2017) pointed out that a dense one-stage detector evaluates tens of thousands of candidate locations per image, and the overwhelming majority are easy background that contributes almost nothing useful to training — yet standard cross-entropy loss weighs every one of those easy negatives the same as the rare, hard, genuinely informative examples. Focal loss down-weights the contribution of well-classified examples so that training gradient comes predominantly from the hard cases, and combined with a Feature Pyramid Network for multi-scale features, RetinaNet was the result that first showed a one-stage detector could match two-stage accuracy rather than trading accuracy for speed.

DETR replaced the hand-engineered proposal-and-suppression pipeline with a single trainable set-prediction problem. Carion and colleagues at Facebook AI (2020) fed CNN features into a Transformer encoder-decoder that directly outputs a fixed-size set of predictions, each either an object or "no object," and matched predictions to ground truth during training using bipartite (Hungarian) matching rather than anchor boxes or hand-tuned IoU thresholds. Because each predicted box is already unique by construction, DETR needs no non-maximum suppression step at all — the duplicate-removal problem that every anchor-based detector has to handle is designed out of the architecture rather than cleaned up afterward. The cost was originally slow convergence and weak performance on small objects, both of which later DETR variants substantially closed.

Read object detection's architectural history as successive attempts to remove a hand-engineered step: Faster R-CNN made the proposal stage learnable instead of a separate algorithm, YOLO and SSD removed the separate proposal stage entirely, focal loss fixed the class imbalance that dense one-stage prediction creates, and DETR finally removed the hand-tuned non-maximum suppression step itself — each generation trading one engineered heuristic for one more piece of the pipeline the network learns on its own.