image segmentation fully convolutional networks

Image segmentation assigns a label to every pixel in an image rather than drawing a box around an object, which is the difference between knowing roughly where a car is and knowing the exact outline of its body down to the mirror. That pixel-level output is three different tasks wearing one name, and which one a system needs changes the entire architecture: semantic segmentation labels every pixel's category without distinguishing individual objects, instance segmentation separates each object into its own mask, and panoptic segmentation does both at once — instance masks for countable "things" like cars and pedestrians, semantic labels for uncountable "stuff" like road and sky.

U-Net proved that an encoder-decoder with skip connections could segment accurately from very little training data. Ronneberger and colleagues (2015) built an architecture that compresses an image down through a contracting encoder path and then expands it back up through a decoder, but the detail that made it work was connecting each encoder layer directly to its matching decoder layer — skip connections that hand the decoder back the fine spatial detail the encoder's pooling had thrown away. Designed originally for biomedical images, where labeled training data is scarce and expensive to produce, U-Net's data efficiency made it the dominant architecture for medical segmentation specifically, and the encoder-decoder-with-skip-connections pattern it introduced now underlies far more than segmentation — it's the same backbone shape used inside the denoising network of most image-generating diffusion models.

DeepLab solved the resolution problem a different way: never fully downsample in the first place. Chen and colleagues at Google (DeepLab v3+, 2018) used dilated — also called atrous — convolutions, which widen a convolution's receptive field by inserting gaps between its kernel weights instead of using pooling to shrink the feature map. That preserves spatial resolution throughout the network while still letting each pixel's prediction draw on a wide context. DeepLab v3+ pairs this with Atrous Spatial Pyramid Pooling, which runs several dilated convolutions at different gap rates in parallel to capture context at multiple scales simultaneously, and the combination reported around 82 percent mean intersection-over-union on the Cityscapes urban driving benchmark — among the strongest semantic segmentation results on that dataset at publication.

Mask R-CNN turned instance segmentation into detection with an extra branch. He and colleagues at Facebook AI Research (2017) extended the Faster R-CNN object detector by adding a parallel branch that predicts a binary mask for each detected box, alongside the existing class and box-coordinate predictions the detector already made. The key fix that made the masks spatially accurate was RoI Align, which replaced Faster R-CNN's coarser region-pooling operation with one that avoids quantizing coordinates to the nearest integer — a small-sounding fix that turned out to matter a great deal once the output is a pixel-precise mask rather than just a bounding box, where the original rounding error was invisible.

The Segment Anything Model replaced per-task training with a single prompt-driven foundation model. Kirillov and colleagues at Meta (2023) trained SAM on SA-1B, a dataset of roughly eleven million images paired with 1.1 billion masks — collected with a model-in-the-loop pipeline where an earlier version of SAM itself proposed masks for humans to correct, generating most of the dataset's annotations automatically rather than by hand. The result accepts a point click, a bounding box, or rough text as a prompt and segments the indicated object without any task-specific fine-tuning, functioning as a general-purpose segmentation tool the way a large language model functions as a general-purpose text tool — one trained model, steered by a prompt, instead of one model trained per task.

Class imbalance is the training obstacle that's specific to dense pixel-level prediction. In almost any natural image, background pixels vastly outnumber the pixels belonging to the object of interest — a single small tumor in a medical scan, or a distant pedestrian in a driving scene, might account for under one percent of the image's pixels. A model trained with plain cross-entropy loss can reach deceptively high pixel accuracy by learning to predict "background" everywhere and barely bothering with the rare foreground class, which is why segmentation training typically uses class-weighted loss or Dice loss — a loss term built directly from the overlap between predicted and true masks — instead of unweighted per-pixel cross-entropy.

Read image segmentation as three related but distinct questions about the same pixel grid — what category is this, which object does it belong to, and both at once — and the architectural history follows directly from how to answer each one efficiently: U-Net's skip connections recover detail pooling destroys, DeepLab's dilated convolutions avoid destroying it in the first place, Mask R-CNN turns instance separation into a branch on a detector, and SAM replaces training a new model per task with prompting one model that was already trained on everything.