epoch
An epoch is one complete pass through the entire training dataset, and how many of them a model gets is one of the sharpest dividing lines between classic deep learning and modern LLM pretraining: the ResNet-50 recipe for ImageNet trains for 90 full epochs over the same 1.28 million images, while a frontier language model typically never finishes a single epoch at all — Meta's Llama 3.1 paper (2024) reports training on 15 trillion tokens, a corpus so large relative to the compute budget that each token is seen once, if that. Inside either regime, the same three nested units apply: the epoch is the outer loop, the batch is a chunk of data processed together, and an iteration (or step) is one weight update from one batch.
The arithmetic behind "epochs" and "iterations" is one division and one multiplication. For ImageNet's 1.28 million training images at a batch size of 256, each epoch requires 5,000 iterations (1,280,000 divided by 256), so the full 90-epoch ResNet-50 schedule runs 450,000 iterations in total — a number large enough that training logs and learning-rate schedulers track steps directly rather than waiting for epoch boundaries to report progress. Nearly every modern scheduler, including the warmup-then-cosine-decay schedules standard in GPT and Llama training, is defined in steps for exactly this reason: a step is the atomic unit of training time, while an epoch is just a convenient way of saying "the dataset has now been covered once."
Gradient accumulation changes what counts as one iteration without changing what counts as one epoch. When GPU memory limits how many examples fit in a single forward-backward pass, several smaller micro-batches can be accumulated before a single optimizer step is taken, so the effective batch size becomes the per-GPU micro-batch size multiplied by the number of GPUs multiplied by the number of accumulation steps — the detail covered in full under batch-size. What matters for epoch counting is that only the optimizer step counts as an iteration; the intermediate micro-batch forward-backward passes that feed into it are bookkeeping, not separate steps, which is why training logs report optimizer steps rather than micro-steps.
Shuffling before each epoch is what keeps "pass number 47" from teaching the model the wrong lesson. Without reshuffling, a model sees batches in the exact same order every epoch, which lets it pick up on spurious patterns tied to data order rather than the signal in the data itself; for LLM pretraining, where documents are concatenated and split into fixed-length sequences before training, the notion of a clean epoch boundary becomes approximate rather than exact, which is one more reason large-scale pretraining is described in tokens processed rather than epochs completed.
Epoch, batch, and iteration describe the same underlying loop at three different grain sizes, and which one a team actually tracks reveals which resource is scarce in their setup: a vision team counting epochs has a fixed, reusable dataset and a comfortable compute budget relative to it, while an LLM team counting tokens processed has a compute budget that runs out long before the dataset does — the vocabulary stayed the same across that shift in scale, but which word actually matters did not.