LoRA and parameter-efficient training
How to adapt a billion-parameter model on a single card: rank decomposition, the memory calculation, and the settings that matter.
The idea
Training a model fully means changing all its billions of weights. LoRA leaves them untouched and places a small learnable correction alongside. Only that correction is trained.
The picture: rather than rewriting a book, you stick margin notes into it.
What that buys
| Full | LoRA | |
|---|---|---|
| Trainable parameters | 100 % | 0.1 to 1 % |
| Memory for a 7 bn model | ~112 GB | ~18 GB |
| Result file | ~14 GB | 20 to 200 MB |
| Several variants | one full model each | one adapter each |
The settings
| Parameter | Meaning | Usual value |
|---|---|---|
r | Rank of the decomposition | 8 to 64 |
alpha | Adapter scaling | usually 2r |
dropout | Regularisation inside the adapter | 0 to 0.1 |
| Target modules | Which layers are adapted | All attention projections, often the feed-forward too |
| Learning rate | 1e-4 to 3e-4 |
- Adapt all four attention projections, not just two. The difference is measurable.
- Start from
alpha = 2r, then vary only one of the two. - Merge the adapter into the weights for deployment when no switching is needed. That saves inference time.
The formula
Worked through
For a matrix with d = k = 4096 and r = 16:
- Full:
4096 × 4096 = 16,777,216parameters - LoRA:
16 × (4096 + 4096) = 131,072parameters - Share: 0.78 percent
Across 32 layers with four such matrices each, that is
32 × 4 × 131,072 = 16.8 million trainable parameters against 2.15 billion. The
adapter occupies around 34 megabytes in float16.
Memory during training
| Item | Full | LoRA | QLoRA |
|---|---|---|---|
| Weights | 14 GB (fp16) | 14 GB (fp16) | 4 GB (nf4) |
| Gradients | 14 GB | 0.03 GB | 0.03 GB |
| Optimiser states | 56 GB (fp32) | 0.13 GB | 0.13 GB |
| Activations | 10 to 30 GB | 6 to 20 GB | 6 to 20 GB |
| Total | ~100 GB | ~24 GB | ~12 GB |
QLoRA additionally quantises the frozen weights to 4 bits and trains the adapter
in bfloat16. That fits adaptation of a 7-billion model onto a 16 GB card.
Why a low rank suffices
The empirical observation is that the change ΔW during fine-tuning has a low
intrinsic dimension: adapting to a narrow task moves within a small subspace. A
rank of 8 to 16 therefore suffices for format and style. For broader changes, such as a new language or a fundamentally different domain,
it does not, and full training is
genuinely superior there.
Serving several adapters
Modern runtimes load one base model plus several adapters and select per request. That is the real operational advantage: one model in memory, arbitrarily many domain variants, each a file of a few megabytes with its own version and its own approval state.
Related courses and sources
LoRA
Adaptation through a few additional parameters instead of full training. The reason fine-tuning is affordable today.
For anyone fine-tuning on a budget; the basis of affordable adaptation.