Vectors and matrices

The idea#

A computer can do nothing with the word "dog". It can compute with the list [0.82; −0.13; 0.44; …]. The whole trick of modern AI is translating things into number lists such that closeness in the list means similarity in meaning.

Such a list is called a vector. A table of many vectors is a matrix. That is all the vocabulary needed to begin.

What it is good for#

Once meaning becomes direction, questions become geometry:

• "Which document matches my question?" becomes "Which vector points almost the same way?"
• "Which word comes next?" becomes "Which vector has the largest dot product with the current state?"
• "Are these two faces the same person?" becomes "Is the distance between the two vectors below a threshold?"

The three calculations you need#

Dot product. Multiply element by element, then add everything up. The result is one number. Large means "pointing the same way".

Norm. The length of a vector: the square root of its dot product with itself. Needed to separate similarity from magnitude.

Matrix multiplication. Every row of the first matrix dotted with every column of the second. This is the one operation a GPU runs tens of thousands of times in parallel.

import numpy as np

a = np.array([0.8, 0.1, 0.6])   # "termination"
b = np.array([0.7, 0.2, 0.5])   # "contract cancellation"
c = np.array([-0.4, 0.9, 0.1])  # "barbecue"

def cosine(x, y):
    # Dividing out the norm is the whole difference between
    # "how similar" and "how similar and how long".
    return float(x @ y / (np.linalg.norm(x) * np.linalg.norm(y)))

print(round(cosine(a, b), 3))   # 0.993  -> nearly the same direction
print(round(cosine(a, c), 3))   # -0.055 -> effectively unrelated

Common mistakes#

What you can measure#

The distribution of cosine values across your own corpus is a good diagnostic. If random document pairs average 0.7, the embedding is too unspecific for that corpus and every search on it will feel arbitrary.

The formulas#

Worked through#

With a = [3, 4] and b = [4, 3]:

• a · b = 3·4 + 4·3 = 24
• ‖a‖ = √(9 + 16) = 5, ‖b‖ = √(16 + 9) = 5
• cos = 24 / 25 = 0.96

An angle of about 16 degrees. For contrast c = [−4, 3]: a · c = −12 + 12 = 0, so cos = 0 and exactly 90 degrees. Orthogonal here means: the two features say nothing about each other.

Cost and memory#

Worked through for a typical layer with n = 1 (one token), k = 4096, m = 16384: 2 × 1 × 4096 × 16384 ≈ 1.34e8 FLOPs. With 32 layers and two such matrices per layer that is roughly 8.6 billion operations per token. Which is exactly why inference is expensive although no individual step is hard.

Memory, just as directly: a 4096 × 16384 matrix has 67.1 million entries. In float16 that is 134 MB, in int8 67 MB, in int4 34 MB. The table in Memory and bandwidth works this through for whole models.

Numerical pitfalls#

Matrix multiplication is not associative in floating point: the order of the additions changes the last bit. Across 4096 summands in float16 this accumulates measurably, which is why libraries accumulate internally in float32 even when the weights are float16. Turn that off and you get models that answer differently on two different cards.