Embeddings
Meaning as direction in space: how words, sentences and images become vectors, what they are used for, and where similarity misleads.
The idea
Every word, sentence and image gets a point in a space of hundreds of dimensions. Things that resemble one another sit close together. The question "what is similar?" thereby becomes a distance measurement.
What it is good for
- Search without exact keywords.
- Finding duplicate entries.
- Grouping and sorting texts.
- The basis of every RAG application.
The kinds
| Kind | What is embedded | Example use |
|---|---|---|
| Word embedding | A word, independent of sentence | Classical text analysis |
| Contextual embedding | A token in its sentence | Inside a language model |
| Sentence or passage embedding | A whole text chunk | Search, RAG, grouping |
| Image embedding | An image | Image search, duplicate detection |
| Joint image-text embedding | Both in one space | Cross-modal search |
import numpy as np
def normalise(v):
# After normalising, the dot product equals cosine similarity.
return v / np.linalg.norm(v, axis=-1, keepdims=True)
E = normalise(np.random.default_rng(0).normal(size=(5, 384)))
query = normalise(np.random.default_rng(1).normal(size=(384,)))
scores = E @ query
print(np.argsort(scores)[::-1][:3], np.sort(scores)[::-1][:3].round(3))- Always normalise before storing or comparing.
- Use the same prefix the model used in training; many models distinguish query and document embeddings.
- Store the model version. A model change invalidates the whole index.
How they are trained
Quality depends heavily on how hard the counterexamples are. Random ones are too easy: the model learns only to separate topics coarsely. The real lever is hard negatives: documents that look similar and are nevertheless wrong.
Dimension as a trade-off
| Dimension | Memory per 1 million (float32) | Typical quality |
|---|---|---|
| 384 | 1.5 GB | Sufficient for most applications |
| 768 | 3.1 GB | Slightly better, the usual standard |
| 1,536 | 6.1 GB | Rarely measurably better |
| 3,072 | 12.3 GB | Only for very large heterogeneous collections |
Methods with nested representations allow a long vector to be truncated at an arbitrary point while keeping most of the quality. That enables two-stage search: coarse with 128 dimensions over the whole collection, fine with 768 over the best thousand.
The data protection point
An embedding is not anonymisation. Two attacks are practical:
- Inversion. A sentence embedding can be approximately inverted to its source text, often almost verbatim for short texts.
- Membership. Proximity to a known vector reveals whether a particular document is in the collection.
In practice that means the vector index carries the same protection duties as the source collection, including erasure and access requests. See Vector databases.
Related courses and sources
Efficient Estimation of Word Representations
The paper that first established words as vectors with computable meaning. The origin of all embeddings.
The origin of all embeddings; short and still illuminating.
Essence of Linear Algebra
Fifteen short films showing what a matrix does to space. If you have only ever seen vectors as lists of numbers, you will see something else afterwards.
For anyone who has only ever seen vectors as lists of numbers and needs an intuition.
Learning Transferable Visual Models
Images and text in one shared space. The basis of searching images by description and of image generation.
For understanding image search by description and image generation.
Retrieval-Augmented Generation
The paper that joined retrieval and generation. The origin of the pattern that makes your own documents usable with citations.
For anyone making their own documents usable; the origin of the pattern.