Embeddings and Retrieval

Turning meaning into vectors, searching by similarity, and grounding LLM answers in your own documents.

Intermediate lesson, about 40 minutes, with interactive demos and a quiz.

What you will learn

Meaning as a position

If meaning is a position, similarity is a direction. The workhorse of every embedding system is a one-line formula from school geometry: the cosine of the angle between two arrows.

What does it mean for meaning to have a position?

Picture a map where every sentence is a point. Sentences about football gather in one region, sentences about baking in another, and a sentence about a footballer’s diet sits somewhere between them. Real embedding spaces have hundreds or thousands of axes instead of two, and no single axis means anything as tidy as “sport” or “food”. What matters is the arrangement: which points are near which.

How do we measure “near”?

Each embedding is an arrow from the origin. The dot product of two arrows multiplies them coordinate by coordinate and adds the results. It is large when the arrows point the same way and are long, zero when they are perpendicular, and negative when they point apart. Divide by both lengths and you get cosine similarity, which depends only on the angle:

cos(a, b) = (a · b) / (|a| |b|) = (a₁b₁ + a₂b₂ + … + aₙbₙ) / (|a| |b|)

Cosine similarity ranges from 1 (same direction) through 0 (perpendicular, unrelated) to -1 (opposite).

A worked example: a = (3, 4) and b = (4, 3). The dot product is 3×4 + 4×3 = 24. Both have length 5. So cos = 24 / 25 = 0.96, an angle of about 16°: very similar. For a = (1, 0) and b = (0, 1) the dot product is 0 and the cosine is 0, however long the arrows are.

With the “same direction, different length” preset the cosine is exactly 1 even though the Euclidean distance is large. That is the point of cosine: a two-line note and a ten-page report on the same subject should count as similar. In practice many embedding models output vectors already scaled to length 1 (the dashed circle), and then cosine, dot product and Euclidean distance all produce the same ranking, which lets vector databases use whichever is fastest.

Why does this matter?

Once similarity is a single number, everything else is engineering. Search is “highest cosine to the query”. Deduplication is “cosine above 0.95”. Clustering is grouping by cosine. Recommendation is “items near the ones you liked”. Classification can be “which label’s description is nearest”, which is how CLIP classifies images zero-shot.

One caution: cosine scores are only comparable within one model. A score of 0.8 from one model does not mean the same as 0.8 from another, and different models’ vectors cannot be mixed in one index. Thresholds such as “treat anything above 0.75 as a match” have to be tuned on your own data.

Key takeaways

  • Cosine similarity is the dot product divided by both lengths: it measures the angle and ignores length.
  • For unit-length vectors, cosine, dot product and Euclidean distance give the same ranking.
  • Similarity scores are only meaningful within one model; tune thresholds on your own data.

How embeddings are learned

Nobody hand-writes the numbers in an embedding. They are learned from an old linguistic insight: you can tell what a word means from the words that keep it company.

Where do the numbers come from?

If “physician” and “doctor” both keep appearing next to “patient”, “prescribed” and “blood test”, a model can infer they are related without ever being told. This is the distributional hypothesis, and every embedding method, from 1990 to today, exploits it in some form. The methods differ in what counts as “company” and in how the statistics are compressed into a short vector.

Counting and compressing: LSA (1990)

Latent semantic analysis builds a big table with one row per document and one column per word, weighted by TF-IDF, then compresses it with the singular value decomposition, keeping only the k strongest patterns of co-occurrence. Words that show up in similar documents end up sharing those patterns. This is what runs in the demos here.

Many neighbours make sense (“rocket” sits next to “launch”, “pad” and “satellite”), but look at “vaccine”: its closest words include “vet” and “dog”, because one of the two vaccine sentences is about a dog at the vet. An embedding knows only what its training text shows it. At the scale of the web that same property is why embeddings absorb social biases present in the text.

Predicting context: word2vec (2013)

Word2vec replaced counting with a small neural network trained to predict a word from its neighbours (or the neighbours from the word) across billions of words of news text. The by-product is a vector for every word, typically 300 numbers. It became famous for analogies: the vector for king, minus man, plus woman, lands closest to queen. Its limit is that every word gets exactly one vector, so “bank” means the same thing next to a river and next to a mortgage.

Embedding whole passages: Sentence-BERT (2019)

Transformers such as BERT read words in context, which solves the “bank” problem, but they were built to compare two sentences by reading them together. That is slow: the Sentence-BERT paper estimated that finding the most similar pair among 10,000 sentences this way takes about 65 hours. Sentence-BERT fine-tuned BERT so that each sentence is encoded once into a single vector and compared by cosine, cutting the same job to about 5 seconds. Nearly every modern text embedding model follows this “encode separately, compare by cosine” design, often called a bi-encoder.

Contrastive training

Modern embedding models are trained on pairs that should match: a question and the passage that answers it, a title and its article, a post and its top reply. For each pair in a batch, the loss rewards a high cosine for the true partner and a low cosine for every other passage in the batch. Written as a formula, it is a softmax over similarities, the same softmax that picks the next word in a language model:

loss = -log( exp(cos(q, p⁺)/τ) / Σⱼ exp(cos(q, pⱼ)/τ) )

τ is a temperature. The sum runs over the true passage p⁺ and the other passages in the batch. Bigger batches give more, harder negatives.

What do current models look like?

Since 2024 the strongest embedding models have been built by starting from a large language model rather than a small encoder, then training contrastively in stages on huge collections of pairs, many of them generated by other LLMs. Examples include Google’s Gemini Embedding, which by default outputs 3,072 numbers per text, and Alibaba’s open-weight Qwen3 Embedding family in 0.6B, 4B and 8B parameter sizes. Newer models also embed images, audio and video into the same space as text, which is the idea behind the multimodal lesson.

Leaderboards change monthly, so rather than memorise a winner, learn to read them: check the retrieval column rather than the overall average, check the languages and domains you care about, and test on a few hundred of your own queries before switching.

LSA (this page)Counts co-occurrence in your corpus, compresses with SVD. Instant, transparent, but blind to word order and to words it has not seen.word2vecOne learned vector per word from billions of words. Captures analogies; one vector per word regardless of context.Modern bi-encoderA transformer reads the whole passage and outputs one vector, trained contrastively on question-passage pairs.

Key takeaways

  • Embeddings are learned from co-occurrence: words and passages that appear in similar contexts get similar vectors.
  • Sentence embeddings encode each text once and compare by cosine, which makes search over millions of passages practical.
  • Modern models are trained contrastively on matching pairs and inherit whatever patterns and biases their training text contains.

Dimensions and projections

Embeddings live in hundreds or thousands of dimensions, and our eyes live in two. Every picture of an embedding space is a projection, and every projection lies a little.

How many dimensions, and why?

More dimensions give a model room to separate more distinctions (topic, tone, language, formality, domain) but cost memory and search time. A million passages at 3,072 float32 numbers each take about 12 GB before any index overhead. Matryoshka representation learning offers a way out: the model is trained so that the first 256 or 768 numbers already form a good, smaller embedding, and you can truncate vectors to fit your budget. The LSA slider in the Embedding Explorer shows the same trade-off in miniature: too few dimensions blur topics together, too many bring back word-for-word matching.

How do we draw them?

PCA (principal component analysis) finds the two directions along which the points vary most and drops a perpendicular shadow onto them. It is linear and honest: distances in the picture are shrunk versions of real distances. But two axes rarely capture more than a fraction of the variance, so clusters overlap in the shadow even when they are well separated in the full space.

t-SNE and UMAP are non-linear. t-SNE converts distances into probabilities of being neighbours, then moves points around in 2D until the 2D neighbour probabilities match. Clusters come out beautifully separated, but cluster sizes and the gaps between clusters mean very little, and the picture changes with the perplexity setting and the random start.

PCA keeps only a small share of the variance in its two axes (the figure above the plot), so the topics smear into each other. t-SNE pulls them into tidy islands, but each re-run arranges the islands differently. Neither changes a single search result: retrieval always uses the full vectors. Use projections to explore and to spot problems (a cluster of duplicates, a topic that is split in two), never as evidence of how similar two groups are.

Explore the same corpora with a similarity matrix, a quality curve over k, and your own documents in the Embedding Explorer lab.

Key takeaways

  • More dimensions separate more distinctions but cost memory; Matryoshka-style models let you truncate vectors.
  • PCA is a faithful linear shadow that usually overlaps clusters; t-SNE and UMAP separate clusters but distort sizes and gaps.
  • Search always runs on the full vectors, so a 2D picture is for exploring, not for measuring.

Retrieval-augmented generation

A language model only knows what was in its training data, frozen at a cutoff date. Retrieval-augmented generation hands it the right pages at the moment it answers.

What is RAG?

Retrieval-augmented generation, named by Patrick Lewis and colleagues in 2020, splits answering a question into two jobs. A retriever finds passages likely to contain the answer; a generator, today almost always a large language model, reads those passages and writes an answer grounded in them. The model does not have to have memorised your company handbook, last week’s release notes or a customer’s contract. It has to read well.

How does the pipeline work?

There are two halves: an offline half that runs whenever documents change, and an online half that runs for every question.

  1. Chunk. Split each document into passages small enough to embed well and to fit several into a prompt, often a few hundred words, ideally along natural boundaries such as headings and paragraphs.
  2. Embed and index. Turn every chunk into a vector and store it with its text and metadata (source, date, permissions).
  3. Retrieve. Embed the question with the same model and fetch the top k chunks, often 20 to 50, filtered by metadata.
  4. Rerank. Optionally, a cross-encoder reads the question and each candidate together and re-scores them. It is slower than the bi-encoder but more accurate, so it is used only on the short list, keeping perhaps the best 5.
  5. Prompt and generate. Put the passages in the prompt with instructions to answer only from them and cite which passage supports each claim.

Try “Where did the idea of RAG come from?” with one sentence per chunk: the answering sentence spells out “retrieval-augmented generation” instead of “RAG”, so other sentences outrank it. With two or three sentences per chunk the answer rides along with its neighbours and moves up. Now ask your own question using words the notes never use: the classical retriever finds nothing to hold on to. A neural embedding model would do much better on paraphrases, but the pipeline, and the way it fails, is identical.

Why does RAG matter?

It is the most common way organisations put LLMs to work on their own information, because it keeps knowledge outside the model. Updating an answer means editing a document and re-embedding it, not retraining. Answers can cite sources, which lets people check them. And access control can be enforced at retrieval time, so a user only ever gets passages they are allowed to see. It is also the memory system behind many AI agents, which retrieve notes, past conversations and tool documentation as they work.

Long context windows have not made retrieval obsolete. Filling a million-token window on every question is slow and expensive, and models do not use every part of a long context equally well (more on that below). Retrieval chooses what deserves the model’s attention.

Hybrid search and contextual chunks

Two refinements are now standard. Hybrid search runs a keyword ranker such as BM25 alongside the embedding search and merges the lists, so exact names and codes are not lost. Contextual chunking fixes chunks that make no sense alone (“Revenue grew 3% over the previous quarter”: which company, which quarter?) by prepending a short summary of where the chunk comes from before embedding it. In Anthropic’s tests, contextual embeddings plus contextual BM25 cut the rate of failed top-20 retrievals from 5.7% to 2.9%, and adding a reranker cut it to 1.9%.

Key takeaways

  • RAG = retrieve relevant passages, then have an LLM answer from them with citations.
  • Chunking, the embedding model, top-k and reranking decide what the model sees; the model cannot use what retrieval missed.
  • Hybrid keyword plus embedding search and contextual chunks are standard, measurable improvements.

Measuring retrieval

When a RAG system gives a wrong answer, the first question is not “is the model smart enough?” but “did it ever see the right passage?” That question has a number.

What do we measure?

Retrieval is evaluated separately from generation, using a set of questions where you know which passages are relevant. The two workhorse metrics are:

recall@k = (relevant passages in the top k) / R MRR = mean of 1 / (rank of first relevant passage)

R is the number of relevant passages for the question; average both over all test questions.

A worked example with three questions, each with one relevant passage, found at ranks 1, 4 and 2. Recall@3 is 2/3 = 0.67 because the second question’s passage is outside the top 3. MRR is (1/1 + 1/4 + 1/2) / 3 = 0.58. Recall asks whether the answer made it into the prompt; MRR rewards putting it first.

Bigger chunks tend to raise recall, because each chunk is more likely to contain the answer, but the context the model has to read grows with them, and each chunk carries more irrelevant text. The question that fails everywhere is a paraphrase: “files are edited but the vectors are not updated” never uses the notes’ words “stale”, “documents” or “embeddings”. That is exactly the kind of failure a neural embedding model or a reranker is bought to fix, and exactly how you would prove the purchase worked.

How do I evaluate my own system?

  • Build a test set from real questions. Fifty to a few hundred questions from logs or from the people who will use the system, each with the passages that answer it. LLMs can help draft questions from your documents, but have people check them.
  • Measure retrieval and answers separately. Recall@k and MRR for the retriever; correctness and faithfulness to the sources for the answers, graded by people or by an LLM judge you have spot-checked.
  • Use public benchmarks to shortlist, not to decide. MTEB ranks embedding models across many tasks, and BEIR showed that models which shine on one dataset can fall below plain BM25 on another domain. Your documents are another domain.
For precision, recall and why a single number can mislead, see Evaluating Models.

Key takeaways

  • Recall@k asks whether the answer reached the prompt; MRR rewards ranking it first.
  • Evaluate retrieval separately from generation, on questions from your own users.
  • Public leaderboards shortlist models; only your own test set decides.

When retrieval goes wrong

Most RAG failures are not the language model’s fault. They happen upstream, in how documents were split, which ones were indexed, and what the retrieved text says.

What goes wrong?

A field guide to the common failures, and the usual fix for each:

Bad chunking

A table split from its header, an answer split across two chunks, or one chunk covering five topics whose vector points at none of them. Fix: split on document structure, add overlap, prepend context.

Stale or incomplete index

A policy was updated but its chunks were not re-embedded, or a new document never made it into the index, so the system confidently quotes last year. Fix: re-index on change, store dates, prefer recent sources.

Vocabulary mismatch

Users say “refund”, documents say “reimbursement”; users paste part numbers the embedding model treats as noise. Fix: hybrid search, query rewriting, domain-tuned embeddings.

Lost in the middle

The right passage is retrieved but buried among twenty others. Models use information at the start and end of a long context better than in the middle. Fix: rerank, send fewer and better passages, put the best first.

Permission leaks

Retrieval ignores who is asking and surfaces a passage from an HR file. Fix: filter by access rights at retrieval time, never only in the prompt.

Prompt injection through retrieved text

A web page or email in the index contains instructions aimed at the model. Retrieval faithfully delivers them into the prompt.

The “lost in the middle” effect was measured by Liu and colleagues, who moved the one relevant document around a long context and found accuracy highest when it came first or last. Long-context models have improved since, but the lesson holds: retrieval quality is about putting the right passage where the model will use it, not just somewhere in the prompt.

Why is prompt injection special?

Every other failure makes answers worse. Injection makes the system work for someone else. Greshake and colleagues showed that instructions planted in content an LLM application retrieves, such as a web page, an email or a shared document, can make it leak data or take actions the user never asked for. An embedding model cannot tell a helpful passage from a malicious one: the attacker’s page is simply relevant.

There is no complete fix yet. Defences are layered: mark retrieved text clearly as data and tell the model not to follow instructions inside it (as the prompt in the RAG demo does); filter what gets indexed; limit which tools the model can call when untrusted text is in its context; and require a human to confirm consequential actions. These matter most for agents, which can act on what they read.

Key takeaways

  • Most failures are upstream: chunking, stale indexes, vocabulary mismatch and passage placement.
  • Retrieved text is untrusted input; indirect prompt injection turns relevance into an attack.
  • Layer defences: treat passages as data, filter by permissions at retrieval, restrict tools and confirm actions.

Check your understanding

Eight situations you could meet building search or RAG. Each one asks you to apply an idea from the lesson.

Question 1 of 8

Your help-centre search returns nothing for “can’t get into my account”, although an article titled “Resetting a forgotten password” exists. Which change most directly fixes this?

References

The papers and articles cited in this lesson, from the original latent semantic analysis work to current embedding models and RAG research. The arXiv versions are free to read. To go hands-on, open the Embedding Explorer and try your own documents.

Sources

  1. [1]

    Indexing by latent semantic analysis

    Deerwester, Dumais, Furnas, Landauer and Harshman, 1990

    Journal of the American Society for Information Science 41(6). Introduced LSA: a truncated SVD of the term-document matrix used for retrieval.

  2. [2]

    Efficient Estimation of Word Representations in Vector Space(opens in a new tab)

    Mikolov, Chen, Corrado and Dean, 2013

    The word2vec paper: CBOW and skip-gram models that learn word vectors by predicting neighbouring words.

  3. [3]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks(opens in a new tab)

    Reimers and Gurevych, 2019

    Fine-tunes BERT so whole sentences map to vectors comparable by cosine similarity. EMNLP 2019.

  4. [4]

    Gemini Embedding: Generalizable Embeddings from Gemini(opens in a new tab)

    Lee et al., 2025

    An embedding model initialised from the Gemini LLM, state of the art on the multilingual MTEB benchmark at release.

  5. [5]

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models(opens in a new tab)

    Zhang et al., 2025

    Open-weight embedding and reranking models at 0.6B, 4B and 8B parameters, trained in several stages with LLM-generated training pairs.

  6. [6]

    Matryoshka Representation Learning(opens in a new tab)

    Kusupati et al., 2022

    Trains embeddings whose prefixes are themselves useful embeddings, so vectors can be truncated to save memory. NeurIPS 2022.

  7. [7]

    Visualizing Data using t-SNE(opens in a new tab)

    van der Maaten and Hinton, 2008

    Journal of Machine Learning Research 9. The t-SNE algorithm used in the projection demo.

  8. [8]

    How to Use t-SNE Effectively(opens in a new tab)

    Wattenberg, Viégas and Johnson, 2016

    Distill article showing how perplexity, cluster sizes and distances in t-SNE plots can mislead.

  9. [9]

    Billion-scale similarity search with GPUs(opens in a new tab)

    Johnson, Douze and Jégou, 2017

    The paper behind the FAISS library, including inverted-file (IVF) indexes with product quantisation.

  10. [10]

    Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs(opens in a new tab)

    Malkov and Yashunin, 2016

    The HNSW index. First posted in 2016, published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2018.

  11. [11]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(opens in a new tab)

    Lewis et al., 2020

    Combines a dense retriever over Wikipedia with a sequence-to-sequence generator. NeurIPS 2020.

  12. [12]

    Introducing Contextual Retrieval(opens in a new tab)

    Anthropic, 2024

    Prepends chunk-specific context before embedding; reports top-20 retrieval failure rates falling from 5.7% to 2.9% with contextual BM25, and to 1.9% with reranking.

  13. [13]

    MTEB: Massive Text Embedding Benchmark(opens in a new tab)

    Muennighoff, Tazi, Magne and Reimers, 2022

    A benchmark covering eight embedding task types, including retrieval, clustering and classification; the basis of the public MTEB leaderboard.

  14. [14]

    BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models(opens in a new tab)

    Thakur et al., 2021

    Eighteen retrieval datasets. Found BM25 a robust baseline that many dense retrievers failed to beat out of domain.

  15. [15]

    Lost in the Middle: How Language Models Use Long Contexts(opens in a new tab)

    Liu et al., 2023

    Shows accuracy is highest when relevant information is at the start or end of the context and drops when it is in the middle.

  16. [16]

    Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection(opens in a new tab)

    Greshake et al., 2023

    Demonstrates attacks where instructions hidden in retrieved content hijack an LLM application.

  17. [17]

    Text Embeddings Reveal (Almost) As Much As Text(opens in a new tab)

    Morris, Kuleshov, Shmatikov and Rush, 2023

    Vec2Text: iteratively reconstructs input text from its embedding, recovering many short inputs exactly. EMNLP 2023.

Related