named-entity-recognition-ner machine learning

Named entity recognition finds the spans of text that refer to a specific person, organization, location, date, or other named thing, and labels each span with its category — turning "TSMC announced record Q3 2024 revenue in Taiwan" into structured facts a database or knowledge graph can actually use, rather than leaving that information locked inside a sentence. The task is formulated as token classification: every word in the input gets tagged, and the tags are decoded back into entity spans, which is a deceptively simple framing that the field spent three decades improving the accuracy of.

BIO tagging turns span-finding into something a per-token classifier can solve. Since an entity like "New York City" spans three tokens, a tagging scheme has to mark where an entity begins and where it continues: B-LOC on "New" (beginning of a location), I-LOC on "York" and "City" (inside the same location), and O on every token that's not part of any entity. This reduces a variable-length span-extraction problem to a fixed per-token labeling problem, which is what made it tractable for first statistical models and later neural ones to learn directly, and the BIO scheme (or its BIOES variant, which adds explicit single-token and end-of-span tags) remains the standard output format even for today's transformer-based taggers.

The architecture history is a steady replacement of hand-crafted features with learned ones. Conditional Random Fields (Lafferty and colleagues, 2001) were the pre-deep-learning standard: a statistical sequence model that predicts a globally consistent tag sequence from hand-engineered features — capitalization, part-of-speech tags, whether a word appears in a gazetteer of known entity names. BiLSTM-CRF models (Huang and colleagues, 2015; Lample and colleagues, 2016) replaced the hand-engineered features with a bidirectional LSTM that reads the sentence in both directions and learns its own features, while keeping the CRF layer on top to still enforce globally consistent tag sequences — a combination that produced a substantial accuracy jump over feature-engineered CRFs and became the standard deep-learning baseline. Fine-tuned BERT with a simple linear classification head over each token's contextual embedding (Devlin and colleagues, 2019) then became the new default, because full-sentence context from a pretrained transformer disambiguates cases a BiLSTM alone handles less reliably — the same word "Apple" tagged correctly as an organization in "Apple acquired the startup" and as not-an-entity in "I ate an apple."

Evaluation is strict in a way that punishes near-misses. NER is scored by entity-level F1, which requires both the span boundary and the entity type to match exactly — tagging "New York" when the correct span was "New York City" counts as entirely wrong, not partially right, even though two of three tokens were tagged correctly. CoNLL-2003, built from Reuters news text with four entity types (person, organization, location, and miscellaneous), has been the standard English benchmark since its release and is still the dataset most new NER methods report results against, which makes long-run comparisons across two decades of architectures unusually clean for an NLP task.

The newest systems trade fine-tuning for prompting, at a real cost in latency and a real gain in flexibility. A model fine-tuned for one fixed set of entity types needs new labeled data and a new training run every time a product needs to recognize a category it wasn't trained on — a new entity type in a legal, medical, or financial domain, say. Prompting a large language model to extract entities in a specified JSON schema handles that case with zero additional training, at the cost of higher per-request latency and cost than a small fine-tuned tagger running locally. GLiNER (2023) sits between the two: a compact model trained to be a generalist entity recognizer that accepts an arbitrary list of entity type names at inference time, aiming for near-LLM flexibility on novel categories without LLM-scale inference cost.

Read named entity recognition as the first layer that converts text from something only a human can query into something a database can: every architecture generation changed how the tagging decision gets made, but the BIO-tagged span is still the same output format a rule-based system from 1998 and a prompted LLM in 2024 both ultimately produce, which is exactly why it remains the standard interface between raw text and every downstream system — a knowledge graph, a search index, a compliance pipeline — that needs structured facts rather than prose.