data labeling

Data labeling turns raw examples into the supervised signal a model trains on, and the single hardest thing to internalize about it is that a label is a measurement, not ground truth — it's produced by a person, a policy, or another model following instructions, and all of those are fallible in different ways. A dataset's quality ceiling is set less by how much data was labeled than by how carefully the labeling schema, instructions, and quality control were designed before the first example was ever annotated.

Inter-annotator agreement exists because disagreement between labelers is information, not noise to be averaged away. Cohen's Kappa (1960) measures how much two annotators agree beyond what random chance alone would produce, which matters because raw percent-agreement is misleading on any task with an imbalanced label distribution — two annotators who both label everything "normal" on a dataset that's 95 percent normal agree 95 percent of the time by doing almost no work at all. Fleiss' Kappa extends the idea to more than two raters. Low agreement on a specific example or category usually means the labeling instructions are ambiguous there, not that the annotators are careless, and the right response to disagreement is almost always to revise the guidelines and re-pilot, not to simply take a majority vote and move on — majority vote erases a disagreement that was often telling you something real about the task's actual ambiguity.

ImageNet proved that crowdsourced labeling, not just better algorithms, could unlock a new scale of model capability. Deng and colleagues (2009) built a dataset of millions of labeled images by distributing the labeling task across Amazon Mechanical Turk workers at a scale no small research team could have hand-labeled, organizing the taxonomy around WordNet's existing category hierarchy so the labeling schema itself didn't have to be invented from scratch. The dataset that resulted became the benchmark that catalyzed the deep learning era in computer vision, and the underlying lesson generalized far beyond images: a sufficiently large, carefully organized labeled dataset can be as decisive to progress as an architectural improvement, which is exactly why labeling infrastructure became its own industry rather than staying an afterthought bolted onto research.

Weak supervision replaced hand-labeling individual examples with hand-writing rules that label many examples at once. Ratner and colleagues' Snorkel (2017) introduced "labeling functions" — simple heuristics, patterns, or existing resources that a domain expert writes once, each noisily voting on a label for every example — and then combined the often-conflicting outputs of many such functions into a single probabilistic label per example, using a generative model that learns how reliable and how correlated each labeling function tends to be. The resulting labels are noisier than careful hand-annotation, but because each labeling function covers thousands of examples instantly rather than one annotator covering one example at a time, weak supervision can produce usable training labels at a scale and speed hand-labeling structurally cannot match — a tradeoff of label precision for label volume, made deliberately rather than by accident.

RLHF turned "label this example" into "rank these outputs," a different and harder task that measures preference rather than correctness. Training a reward model from human feedback, as in Ouyang and colleagues' InstructGPT (2022), asks annotators to compare several model outputs for the same prompt and rank them by quality rather than apply a fixed label from a predefined schema — there often isn't a single correct answer, only a relative judgment about which response is better. That relative-ranking format turns out to be both more reliable to collect (people are more consistent comparing two things than scoring one thing on an absolute scale) and directly usable as training signal for a reward model, which is why preference ranking rather than classification became the dominant labeling format for aligning instruction-following models.

Annotation budget is best spent on information, not on volume for its own sake. Active learning formalizes this by having the model currently being trained flag which unlabeled examples it's least confident about, and sending exactly those examples to human annotators next rather than labeling the dataset in arbitrary order — an example the model already classifies confidently and correctly teaches it almost nothing new, while an example near the model's decision boundary is disproportionately informative per label collected. The same logic applies to gold-standard quality checks: a small number of carefully chosen hidden test cases, mixed invisibly into the regular task queue, measures annotator accuracy continuously without the overhead of a separate review pass for every submitted label.

Read data labeling as the step that defines what a model is actually being asked to learn, not a clerical task that happens before the real work starts: the schema and the instructions are a specification, disagreement between annotators is a signal about where that specification is still ambiguous, and every major shift in how labeling is done — crowdsourcing at ImageNet's scale, weak supervision's labeling functions, active learning's uncertainty sampling, RLHF's preference ranking — exists because hand-labeling every example with equal care, in arbitrary order, was never the efficient way to build a dataset.