zero shot learning
Zero-shot learning is a model performing a task or recognizing a category it was never explicitly trained to recognize, with zero labeled examples of that specific target provided. OpenAI's CLIP, described in Radford and colleagues' 2021 paper, is the canonical demonstration: trained on 400 million image-text pairs scraped from the internet using a contrastive objective that pulls matching image-and-caption embeddings together and pushes mismatched pairs apart, CLIP can then classify an image into any set of categories a user names at test time — simply by comparing the image's embedding against the text embeddings of each candidate category name — and the paper reported this zero-shot approach matching a ResNet-50 trained with full supervision on ImageNet's 1.28 million labeled images, despite CLIP never training on a single one of them.
CLIP and GPT-style zero-shot prompting are genuinely different mechanisms that happen to share the same name. CLIP's zero-shot classification works by embedding comparison — the model never generates anything, it only measures distances in a learned embedding space between an image and a set of candidate text descriptions; GPT-3, as described in Brown and colleagues' 2020 "Language Models are Few-Shot Learners" paper, performs zero-shot tasks a completely different way, by taking a natural-language instruction directly in the prompt — "classify this review as positive or negative" — and generating an answer through ordinary next-token-prediction, with no embedding similarity computation involved at all. Both get called zero-shot because neither sees labeled examples of the specific target task during training, but the underlying mechanism producing that behavior is not the same.
Zero-shot accuracy is almost always below what a few-shot or fully fine-tuned model reaches on that same specific task, and the gap is the real price of flexibility. A model fine-tuned on thousands of labeled examples of exactly the task at hand has seen the actual distribution of inputs and outputs for that task; a zero-shot model is generalizing from broader pretraining knowledge and has never adjusted a single weight for the specific target, so it predictably underperforms a specialized model on narrow, well-defined tasks — the trade zero-shot approaches make is accepting that accuracy gap in exchange for never needing labeled data for every new category or task that comes up.
How a category or instruction is phrased measurably changes zero-shot accuracy, which is exactly why CLIP's own paper reports results using prompt templates rather than bare category names. Radford and colleagues found that embedding the phrase "a photo of a dog" rather than the bare word "dog" measurably improved CLIP's zero-shot accuracy, because the model's training data consisted of full captions rather than single words, so a caption-shaped query matches the embedding space CLIP actually learned far better than an isolated noun does — a reminder that zero-shot performance isn't just a property of the model, it's sensitive to how the task gets described to it.
Zero-shot learning's real appeal is the cold-start problem it solves: a system that needs to handle a brand-new category or task the moment it's defined, without waiting for anyone to collect and label examples first, which is precisely the situation CLIP's category names and GPT's prompted instructions both handle — at some cost in accuracy compared to a model that got to train specifically on the thing being asked of it.