cross validation

Cross-validation is the fix for a single train/test split's biggest weakness — that whichever 20% of the data happens to land in the test set determines the reported score, so a model can look great or mediocre purely from the luck of the split. Instead of holding out one fixed slice, K-fold cross-validation rotates the held-out portion: the data is divided into K equal folds, the model is trained K separate times with a different fold excluded each time, and the K resulting scores are averaged into a single, far more stable estimate. The method traces back to Stone's 1974 paper on cross-validatory choice and Geisser's 1975 predictive-sample-reuse work, and Kohavi's 1995 empirical study — still widely cited — is the reason 10-fold stratified cross-validation became the field's long-standing default rather than 5-fold or leave-one-out.

Every data point gets exactly one turn as a test example, never training and testing at once. This is what makes the K-fold average more trustworthy than a single split: with K=5, each of the five scores comes from a model that never saw that particular fold during training, so the final average uses all N examples for testing overall while no individual score mixes train and test data. The standard deviation across the K scores is as informative as the mean — a model scoring 85% ± 1% is far more reliable than one scoring 85% ± 12%, even though both report the identical average.

K is a bias-variance trade-off, not a free parameter to maximize. Leave-one-out cross-validation (K equal to the sample size) uses almost all the data for training on every round, giving a nearly unbiased estimate, but it requires N full training runs and the N estimates are highly correlated with each other, which inflates variance in the final average; Hastie, Tibshirani, and Friedman's *The Elements of Statistical Learning* recommends 5 or 10 as the practical sweet spot for exactly this reason, and scikit-learn's `cross_val_score` and `GridSearchCV` both default to `cv=5`.

Stratified, time-series, and group variants fix a specific way random folding breaks. Plain random K-fold assumes every fold is an interchangeable random sample, which fails whenever the data has structure the split needs to respect — an imbalanced classification problem needs stratified folds that preserve each class's proportion, a forecasting problem needs a time-series split so no fold trains on data from after the test period, and clustered data such as repeated patient visits needs group K-fold so the same patient never appears in both the training and test fold of a round. Each of those is a real, separately useful technique with its own failure modes worth its own explanation; this entry is the general mechanism all of them specialize.

Cross-validation estimates generalization; it does not, by itself, prevent leakage. Scaling the data, selecting features, or tuning a tokenizer using the full dataset before folding lets information from the test fold leak into training through those shared preprocessing statistics, which is why the correct pattern fits every preprocessing step — scalers, feature selectors, encoders — inside each fold via a pipeline, never outside the cross-validation loop. The discipline cross-validation enforces is the same one a single held-out validation set enforces: the examples used to judge a model have to be the one thing that never influenced how the model was built.