loss function

A loss function is the single number a model's entire training process is trying to minimize — it measures how wrong a prediction is, and every weight update during training exists purely to make that number smaller on the next attempt. Choosing the loss function is one of the most consequential decisions in building a model, because it's the literal mathematical definition of what "better" means for that specific task; a model doesn't get smarter in some abstract sense during training, it just gets better at whatever its loss function happens to measure. At a high level, loss sits at the center of every training step:

Classification and regression use fundamentally different loss functions because they're measuring fundamentally different kinds of wrongness. Cross-entropy loss is the standard for classification: it compares the model's predicted probability for the correct class against 1.0, and the penalty grows sharply as that predicted probability drops, which is exactly the behavior you want — being 90% confident in the wrong answer should cost far more than being 51% confident in the wrong answer. Mean squared error is the standard for regression: it measures the squared numeric distance between a predicted number and the true value, which makes large errors disproportionately costly (a single prediction that's off by 10 contributes as much loss as 100 predictions each off by 1), a property that's sometimes exactly right and sometimes exactly wrong depending on whether big misses are genuinely worse in the real task.

Specialized loss functions exist to directly encode a specific priority the standard options don't capture, and that's a real engineering lever, not an academic refinement. Focal loss down-weights the contribution of examples a model already gets right easily, forcing training attention onto the hard, rare cases — exactly the fix needed when one class vastly outnumbers another, such as detecting a rare but critical wafer defect pattern buried in overwhelmingly normal inspection images. Label smoothing softens the target from a hard 100%-confident answer to something like 90% on the correct class and a small amount spread across the rest, which prevents a model from becoming overconfident and tends to improve how well it generalizes to data it hasn't seen.

Auxiliary losses let a single training run optimize for more than one objective at once, which is how modern multi-stage training pipelines actually work in practice. A mixture-of-experts model adds a load-balancing loss term specifically to discourage the routing network from sending all tokens to the same few experts, which would otherwise waste most of the model's total capacity; RLHF's reward-model training stage uses a contrastive loss that pushes a chosen (human-preferred) response's score above a rejected response's score, which is a completely different kind of comparison from ordinary classification or regression loss, built specifically for learning from paired preference data rather than from a single ground-truth label.

Read the loss function as the actual, literal objective a model is being trained toward — not a technical detail to default on, but the mathematical statement of what "good" means for that specific task: change the loss function and, with everything else held constant, you change what the finished model is actually good at, which is why picking the right one is as consequential as any architectural choice in the entire pipeline.