semantic search

Semantic search ranks results by inferred meaning rather than exact word overlap, so a query for "how to reset a forgotten password" can retrieve a document titled "account recovery steps" even though the two share almost no words. Classic keyword search, built on term-frequency methods like BM25 (Robertson and Walker's 1994 Okapi weighting scheme, still the default ranking function in Elasticsearch and OpenSearch), scores a document by how often and how distinctively the query's exact terms appear in it — which is precise for identifiers and named entities but blind to paraphrase. Semantic search instead encodes both query and document into the same embedding space with a model and ranks by vector similarity, trading exact-term precision for the ability to match concepts expressed in different words.

The dominant architecture is a bi-encoder: one model, two independent passes. A bi-encoder, the approach popularized by Sentence-BERT (Reimers and Gurevych, 2019) and Facebook's Dense Passage Retrieval (Karpukhin et al., 2020), encodes the query and every document into fixed-length vectors separately, so all document vectors can be computed once, stored in an index, and reused across every future query. Similarity is then just a vector-space distance — cosine similarity or dot product — computed against millions of pre-encoded vectors through an approximate nearest-neighbor index. The independence is also the limitation: because the model never sees the query and a specific document together, a bi-encoder cannot model fine-grained interaction between them, which is why it is fast enough to be the first stage but not precise enough to be the only stage.

Cross-encoders fix the interaction problem by giving up the ability to precompute. A cross-encoder feeds the query and a single candidate document into the model together, so every token of one can attend to every token of the other, producing a far more accurate relevance score — on the MS MARCO passage-ranking benchmark, cross-encoder rerankers consistently outperform bi-encoder retrieval alone by several points of MRR@10. The cost is that nothing can be precomputed: scoring n candidates means n full forward passes through the model, which is far too slow to run against an entire corpus. The standard pipeline therefore runs a cheap bi-encoder (plus often a sparse method) to pull back a few hundred candidates, then a cross-encoder reranks only that shortlist — buying most of the cross-encoder's accuracy at a small fraction of its cost. ColBERT (Khattab and Zaharia, 2020) occupies a middle position, computing token-level embeddings for documents in advance and deferring only a lightweight interaction step to query time.

Hybrid search exists because dense and sparse methods fail in complementary ways. A dense embedding tends to blur exact identifiers — a part number, an error code, a person's name — because the model was trained to generalize meaning, not memorize tokens; a sparse method handles those perfectly but cannot see that "car" and "automobile" mean the same thing. Reciprocal rank fusion (Cormack, Clarke, and Buettcher, 2009) combines the two ranked lists by each document's rank position rather than its raw score, which sidesteps the problem that a BM25 score and a cosine similarity are not on comparable scales. Most production search systems built after roughly 2021 — Elasticsearch and OpenSearch both added native dense-vector fields and hybrid query support around then — run dense and sparse retrieval together by default rather than choosing one.

Retrieval quality is measured against a shortlist, not a single right answer. Recall@k asks whether the correct document appears anywhere in the top k results; MRR (mean reciprocal rank) rewards finding it early rather than merely finding it; nDCG weighs multiple relevant documents by graded relevance rather than treating retrieval as binary. BEIR (Thakur and colleagues, 2021), a benchmark spanning eighteen retrieval datasets across domains, exists specifically because a dense retriever tuned on one dataset's distribution often generalizes poorly to another — a model that wins on MS MARCO passage ranking can underperform BM25 on a biomedical or legal corpus it never saw in training, which is why hybrid retrieval is the safer default rather than a dense-only bet.

Chunking decides what semantic search can even find. An embedding model has a fixed input length, so long documents have to be split into chunks before encoding, and the split boundary directly shapes recall: a chunk cut mid-answer can bury the relevant sentence between two embeddings, neither of which scores well against the query. Overlapping chunk windows and including surrounding title or section context in each chunk are the two most common mitigations, and the right chunk size is itself domain-specific rather than a fixed constant — short for FAQ-style content, longer for narrative or legal text where meaning depends on surrounding sentences.

Semantic search is best read as a ranking problem with a geometry attached: once meaning is represented as a vector, "relevant" becomes "nearby," and the entire engineering problem is making that nearness fast to compute at scale, accurate enough that nearby really does mean relevant, and robust enough that the rare exact term a dense model blurs still gets found by whatever runs alongside it.