data augmentation

Data augmentation creates new training examples from existing ones through label-preserving transformations — a flipped cat is still a cat, a back-translated sentence still means the same thing — and it works by denying the model the shortcut of memorizing specific pixel patterns or word sequences. If the same underlying concept keeps appearing in slightly different form across training, the only way for the model to fit all of those variations well is to learn the actual invariant feature rather than the particular arrangement of one example. AlexNet (Krizhevsky and colleagues, 2012) already used random crops, horizontal flips, and color jitter specifically to fight overfitting on ImageNet, and augmentation has been a default component of nearly every vision and language training pipeline since.

Mixup does something structurally different from ordinary augmentation: it blends labels, not just inputs. Zhang and colleagues (2017) generate new training examples by linearly interpolating two images pixel by pixel and interpolating their one-hot labels by the same mixing ratio — a 70/30 blend of a cat and a dog image is trained against a label that is 70 percent cat, 30 percent dog, not a hard assignment to either class. This forces the model to output genuinely calibrated, smoothly varying confidence between classes rather than the extreme, overconfident predictions a model trained only on hard labels tends toward, and the soft-label mechanism is the part that distinguishes mixup from augmentations that only ever transform the input.

CutMix combined Cutout's regional masking with mixup's soft labels, and it beat both of its parents on the benchmarks that mattered most. DeVries and Taylor's Cutout (2017) randomly zeroed out square patches of an image, forcing a model to not rely on any single region to make its prediction, but a zeroed-out patch is wasted pixels that carry no training signal at all. Yun and colleagues' CutMix (2019) replaced the zeroed-out patch with a patch cut from a second training image instead, and mixed the two images' labels in proportion to the area each one occupies in the final image — no training signal is wasted, the model has to localize which parts of the image belong to which class, and CutMix outperformed both mixup and Cutout individually on standard image classification and object detection benchmarks of the time.

AutoAugment proved the augmentation policy itself could be learned, and RandAugment proved that learning it was mostly unnecessary. Cubuk and colleagues at Google (AutoAugment, 2018) used reinforcement learning to search over thousands of candidate sequences of augmentation operations and their magnitudes, finding policies specific to a given dataset that improved on hand-picked augmentations — but the search itself cost thousands of GPU-hours, far more compute than training the target model. RandAugment (Cubuk and colleagues, 2019) replaced the massive search space with just two hyperparameters — how many transformations to apply, and how strong each one should be — chosen by a small grid search instead of reinforcement learning, and came within a small margin of AutoAugment's accuracy at a tiny fraction of the search cost, which made learned-style augmentation practical for ordinary training runs instead of only well-resourced research labs.

Text augmentation has to work around the fact that most transformations aren't obviously label-preserving the way an image flip is. Wei and Zou's Easy Data Augmentation (2019) uses four simple word-level operations — synonym replacement, random insertion, random swap, and random deletion — each individually small enough to rarely change a sentence's meaning or label, while still perturbing the exact wording the model would otherwise memorize. Back-translation produces more natural-sounding variation by round-tripping a sentence through another language and back, relying on the asymmetry of translation to introduce paraphrase-level diversity that word-substitution operations can't reach on their own, at the cost of needing a translation model rather than just a word list.

Augmentation strength is itself a hyperparameter with a real failure mode on both ends. Too little augmentation leaves the overfitting problem largely unaddressed; too much and the transformed examples stop resembling real data the model will actually see at inference, effectively training it on a distribution shift of the augmentation's own making — a face rotated so far it no longer looks like a face teaches the model nothing useful. The standard safeguard is evaluating on a held-out set that was never augmented, which is the only way to tell whether a given augmentation strength is actually helping generalization or just adding noise the model has learned to tolerate without it translating to real-world robustness.

Read data augmentation as manufacturing the invariances a model should have learned from more real data: every technique, from a simple flip to mixup's soft-label blending to RandAugment's near-free policy search, exists because the alternative — collecting and labeling enough real data to cover that same variation naturally — is almost always far more expensive than synthesizing it from what's already labeled.