training

Training is the process of adjusting a model's parameters so its predictions get closer to correct, repeated over and over against a dataset until the model converges to a useful set of weights. It's a one-time (or occasional) cost that produces a reusable artifact — the trained model — which is the opposite economic shape from inference, which runs continuously on every request once the model is deployed. At a high level, one pass through training looks like this:

Training cost scales with three multiplied-together factors — model size, dataset size, and how many times the optimizer updates — which is why frontier-model training budgets reached hundreds of millions of dollars even though each individual gradient step is cheap. Epoch AI's public tracking shows frontier training runs now land in the 10^25-10^26 FLOPs range, requiring thousands of GPUs (NVIDIA H100 or B200-class) running continuously for weeks to months; that compute bill, not the conceptual difficulty of the algorithm, is the real reason only a small number of organizations worldwide can train a frontier-scale model from scratch, and why most companies building AI products train nothing from zero and instead fine-tune an already-pretrained model.

The optimizer and learning-rate schedule are what actually decide whether a training run converges to a useful model or wastes its entire compute budget. Plain gradient descent with a fixed learning rate reliably fails at modern scale; Adam and AdamW track per-parameter momentum and adaptive scaling, and virtually every large model trains with a warmup phase (ramping the learning rate up gradually to avoid early instability) followed by a decay schedule (cosine or linear, to let the model settle precisely as it approaches a good solution) — getting this schedule wrong is one of the most common ways an expensive training run produces a mediocre model despite a sound architecture and clean data.

Distributed training exists purely because no single chip holds enough memory or compute for a frontier model, and the specific parallelism strategy chosen directly shapes the hardware a training cluster needs. Data parallelism replicates the full model across many GPUs, each processing a different batch of data and periodically synchronizing gradients — simple to implement, but requires fast interconnects (NVLink, InfiniBand) because every GPU has to exchange gradient updates with every other GPU on a regular cadence. Model and pipeline parallelism instead split a single model's layers or parameters across multiple GPUs, used when a model is too large to fit on one device at all, which trades communication pattern complexity for solving the memory-capacity problem data parallelism alone can't.

Read training as the capital expenditure in AI's economics and inference as the ongoing operating expense: training happens rarely and produces a durable asset (the model), inference happens continuously and is billed per request forever, and nearly every hard engineering problem in modern AI infrastructure — distributed parallelism strategies, optimizer schedules, chip interconnect bandwidth — exists specifically to make one or the other of those two very differently shaped costs affordable at the scale the industry now operates at.