Diffusion models
The second large model family alongside transformers: denoising rather than continuing, why that suits images better, and where the two meet.
The idea
Two directions. Forward: take an image and add noise step by step until nothing is recognisable. That path is trivial and needs no model. Backward: learn to undo one of those steps.
If you can undo one step you can undo them all. So you start at pure noise and denoise repeatedly until an image stands there.
The difference from a language model
| Language model | Diffusion model | |
|---|---|---|
| Produces | one token after another | everything at once, refined step by step |
| Step count | as many as there are tokens | fixed, usually 20 to 50 |
| Suited to | sequences with an order | images, audio, anything without a fixed order |
| Steering | prompt in the context | prompt as a condition at every step |
The noise schedule
How quickly noise increases is a design decision with a noticeable effect.
| Schedule | Shape | Property |
|---|---|---|
| Linear | uniform | Simple, the original variant |
| Cosine | slow at first, faster later | Better detail fidelity |
| Sigmoid | soft at both ends | For high resolutions |
The reason for cosine: with a linear schedule the image is near pure noise after a third of the steps, and the remaining two thirds contribute little to learning.
The solvers
| Solver | Steps | Note |
|---|---|---|
| DDPM | 1,000 | Original, stochastic |
| DDIM | 20 to 100 | Deterministic, same seed same image |
| DPM-Solver++ | 15 to 30 | The usual choice today |
| Distilled | 1 to 4 | A model learns to take several steps at once |
Forward and backward
DDIM's determinism matters practically: the same seed and the same condition give exactly the same image, which is what makes reproducibility and targeted variation possible at all.
Why latent
Diffusion on pixels at 1024 by 1024 by 3 means 3.1 million dimensions per step, and there are twenty to fifty steps. A preceding autoencoder compresses by a factor of 8 per side:
| Level | Dimensions | Ratio |
|---|---|---|
| Pixel space 1024² × 3 | 3,145,728 | 1 |
| Latent space 128² × 4 | 65,536 | 1/48 |
Compute per step falls accordingly at barely visible quality cost. That is why image generation runs on a card with 8 GB.
Where diffusion and transformers meet
Current image models replace the once-standard U-Net with a transformer over image patches. The diffusion process is unchanged; only the network predicting the denoising step is a transformer. The same scaling properties as for language models then apply, more parameters and more data improve results predictably, which was less clearly true for U-Nets.
Related courses and sources
Denoising Diffusion Probabilistic Models
Removing noise step by step instead of generating an image in one pass. The basis of every current image model.
For understanding every current image model, thought through from noise.
High-Resolution Image Synthesis
Diffusion in a smaller latent space rather than in pixels. The step that made image generation possible on ordinary hardware.
For anyone running image generation themselves; explains why it works on ordinary hardware.
Understanding Deep Learning
A modern textbook with unusually clear figures that already covers transformers and diffusion models in full. Free as a PDF.
For a modern entry point; it already covers transformers and diffusion in full.