sentiment analysis

Sentiment analysis classifies text by the attitude it expresses — positive, negative, or neutral, and often finer grades than that — and every generation of methods for doing it has been a response to the same handful of failure cases: negation flips polarity without changing most of the words, sarcasm inverts the literal meaning entirely, and a single sentence can carry different sentiment toward different things it mentions. A method that only counts positive and negative words gets all three wrong; the field's history is the story of fixing that one limitation at a time.

The benchmark that made fine-grained sentiment measurable is a tree, not a label. The Stanford Sentiment Treebank (Socher and colleagues, 2013) didn't just label whole movie reviews positive or negative — it labeled every phrase in every sentence's parse tree with one of five sentiment classes, from very negative to very positive, which let models be trained and evaluated on how sentiment compositionally builds up from sub-phrases rather than only on the final whole-sentence label. SST-2, the binary-collapsed version of that dataset, became one of the standard tasks in the GLUE benchmark suite, and fine-tuned BERT-class models reach into the mid-90s percent accuracy on it — a level that plain lexicon scoring, which has no way to learn from labeled examples at all, has never been able to approach.

Negation is cheap to break and expensive to fix properly. "Not bad" is a mildly positive statement built from a negative word, and a lexicon that simply sums per-word polarity scores will get it backwards unless someone has hand-coded a rule specifically for negation windows. The deeper problem is that negation doesn't just flip a label, it modulates it — "not bad" and "not terrible" and "not amazing" each shift sentiment by a different amount depending on the word being negated — which is exactly the kind of context-sensitive composition a model that reads the whole sentence (rather than scoring words independently) is built to capture, and precisely what a bag-of-words lexicon approach structurally cannot.

Sentiment toward one entity is not sentiment toward the sentence. "The food was excellent but the service was unbearably slow" expresses two opposite sentiments at once, each attached to a different aspect of the same experience, and a single document-level sentiment label averages or collapses that distinction away. Aspect-based sentiment analysis instead extracts each aspect mentioned in the text and assigns it its own polarity, which is the only representation that's actually useful for a business trying to learn specifically what's working and what isn't from a pile of reviews rather than just whether reviews skew positive on average.

Sarcasm is the hardest case because it requires knowing what's normally true, not just what the words say. "What a great day to be stuck in traffic" is lexically positive — "great day" — and semantically negative, and detecting that gap requires a kind of world knowledge no lexicon or shallow classifier has: that being stuck in traffic is not, in fact, typically a great day. Large language models handle a meaningful fraction of sarcasm correctly precisely because their training exposes them to enough of that common-sense context to recognize the mismatch between what's literally said and what's plausible, which is a genuinely different capability than pattern-matching on sentiment-laden vocabulary.

Zero-shot LLM prompting has eroded the case for training a dedicated sentiment classifier at all, for many use cases. Where fine-tuning BERT on a labeled dataset used to be the standard approach, a general-purpose instruction-following model can now often match or approach that accuracy with nothing but a well-written prompt and no training data — a meaningful shift for anyone without a labeled dataset of their own, at the cost of higher per-query latency and expense than a small fine-tuned model running locally. Domain-specific fine-tuned models still earn their keep where volume is high enough to amortize training cost or where a domain's vocabulary is unusual enough that a general model needs adaptation — FinBERT for financial text and BioBERT for biomedical text are both standard examples of fine-tuning winning over zero-shot precisely because the domain vocabulary carries sentiment signals a general-purpose model wasn't trained to weigh correctly.

Read sentiment analysis as an ongoing negotiation between how much context a method can use and how much labeled data or compute that context costs: lexicons use no context and no data, fine-tuned transformers use full-sentence context bought with labeled training data, and zero-shot LLMs use the broadest context of all — a model's whole prior knowledge of the world — bought instead with inference cost per query, which is why the right choice has never been a single best method, only the cheapest one that still gets the negation, the aspects, and the sarcasm right for a given volume of text.