How a GPU is built
Compute units, warps, tensor cores and the memory hierarchy: what actually happens when a matrix multiplication runs on a graphics card.
The idea
Picture a factory floor with a hundred work crews. Each crew has 32 people all performing the same movement at the same moment, each on a different workpiece. Anyone stepping out of line holds up the whole crew.
That is how a graphics card works. The design explains both the enormous performance on uniform tasks and the total collapse on branch-heavy code.
The levels
| Level | Scope | Analogy |
|---|---|---|
| Thread | one calculation | a worker |
| Warp | 32 threads in lockstep | a crew |
| Block | up to 1,024 threads with shared memory | a shift at one bench |
| Compute unit (SM) | runs many blocks | a workbench |
| Card | 60 to 150 compute units | the floor |
The memory hierarchy
This is the real key to performance.
| Level | Size | Latency | Bandwidth |
|---|---|---|---|
| Registers | 256 KB per compute unit | ~1 cycle | very high |
| Shared memory | 100 to 228 KB per unit | ~20 cycles | high |
| L2 cache | 40 to 100 MB | ~200 cycles | medium |
| Card memory (HBM) | 24 to 192 GB | ~400 cycles | 1 to 8 TB/s |
| Host memory over PCIe | any | ~10,000 cycles | 32 to 64 GB/s |
The jump from the fourth to the fifth row is why a model that does not fit into card memory does not get slightly slower but 30 to 100 times slower.
Divergence
An if inside a kernel is therefore not free but costs up to a factor of 32 in
the worst case. In practice kernels are written so that all threads in a warp take
the same path, for instance by sorting the data beforehand.
Tiling a matrix multiplication
The naive approach reads a whole row and a whole column from card memory for each
output element. At n = 4096 that is 2 · n³ = 1.4e11 reads. The tiled variant
loads T × T blocks into shared memory and uses each loaded element T times:
At T = 64 that is 64 times less memory traffic. This optimisation is exactly
what sits inside cuBLAS and CUTLASS, and it is why a hand-written naive matrix
multiplication typically reaches under 5 percent of card performance.
Tensor cores and matching formats
Tensor cores compute fixed-size blocks, say 16 × 16 × 16, in one instruction. Three conditions must hold for them to engage:
- The data format fits:
float16,bfloat16,int8,fp8ortf32. Plainfloat32does not use them. - Matrix dimensions are multiples of the block size, usually 8 or 16. A width of 4095 rather than 4096 costs noticeably.
- Memory is suitably aligned, which libraries usually ensure.
The second condition explains why model widths in practice are always round numbers like 4096, 5120 or 8192.
Measuring utilisation
High occupancy of the compute units is not the goal; a high share of actually used compute is. The informative quantities are the fraction of peak compute achieved and the fraction of peak bandwidth achieved. If the first is 8 percent and the second 85 percent, the task is memory-bound and any further compute is useless. See Memory and bandwidth.