dropout

Dropout is the specific regularization technique of randomly zeroing a fraction of a layer's activations on every training step, forcing the network to never depend on any single neuron being present. Srivastava, Hinton, and colleagues introduced it in their 2014 JMLR paper, but the technique had already proven itself two years earlier: Krizhevsky, Sutskever, and Hinton's AlexNet used dropout at a rate of 0.5 in its two fully-connected layers and credited it as one of the keys to winning the 2012 ImageNet competition by a margin so large it restarted the entire field's interest in deep learning. The mechanism runs identically on every forward pass during training, and stops entirely at inference.

Inverted dropout's rescaling step exists so inference needs no special handling at all. If a unit survives with probability 1 minus p during training, multiplying every surviving activation by 1 divided by (1 minus p) keeps the expected output of the layer the same whether or not dropout is active — which is precisely why the clean rule "just turn dropout off at test time" works without any separate calibration step. Get the direction of that scaling backwards, or forget to rescale at all, and a model's outputs shift systematically between training and evaluation in a way that looks like a bug in validation loss rather than what it actually is: a missing rescale.

Modern frontier LLMs often use little or no dropout, and that is not a contradiction of the technique — it is what the technique predicts. OpenAI's GPT-3 (2020) used residual and attention dropout of 0.1, a modest rate; but Chinchilla, PaLM, and the Llama family train with dropout at or near zero, because dropout exists to prevent overfitting on a dataset a model could otherwise memorize, and a model trained on trillions of tokens for less than one epoch — see epoch — never gets the repeated exposure to any one example that memorization requires. Dropout's usefulness scales inversely with how large the training corpus is relative to the model, which is why it is standard in a vision model trained for 90 epochs on 1.28 million ImageNet images and nearly absent in a language model trained once over 15 trillion tokens.

Leaving dropout switched on at inference time, rather than off, turns it into an uncertainty estimator. Gal and Ghahramani's 2016 Monte Carlo Dropout technique runs several stochastic forward passes through the same input with dropout still active, producing a different output each time; the spread across those outputs becomes a usable estimate of the model's uncertainty on that particular input, obtained without training any separate uncertainty model or ensemble. It is a repurposing of the same random-masking mechanism for a second job entirely unrelated to the regularization it was invented for.

Dropout's real contribution was showing that a form of ensembling — normally expensive, requiring many separately trained models — could be approximated for free inside a single network's training run, by letting that one network implicitly stand in for an exponential number of smaller ones. Whether a given model should use it at all comes down to one question that has nothing to do with the dropout rate itself: does this model, trained on this amount of data for this many epochs, actually have the opportunity to memorize — because dropout only has something to fix when that opportunity exists.