Fine-tuning in depth
When training your own is worth it, how much data it takes, and why most projects are better served by prompting and retrieval.
The decision
| Problem | Right answer |
|---|---|
| The model does not know our products | Retrieval, not fine-tuning |
| The model answers in the wrong format | Prompt first, then fine-tuning |
| The model misses our tone | Fine-tuning |
| The model does not know our terminology | Glossary in the prompt first, then fine-tuning |
| The model is too slow or expensive | Fine-tune a smaller model |
| The answers are factually wrong | Retrieval with citation duty |
The dataset
- Every example is what the output should look like. A mediocre example teaches mediocrity.
- Variety in the input, consistency in the form of the output.
- Include examples where the correct answer is a refusal. Otherwise the model learns to always answer.
- A held-out evaluation set fixed before training.
- Remove or pseudonymise personal data before training. A model can reproduce its training data.
The orders of magnitude
| Goal | Examples | Effort with LoRA |
|---|---|---|
| Fixed output format | 200 to 500 | a few GPU hours |
| Tone and style | 500 to 2,000 | a few GPU hours |
| Domain language | 2,000 to 20,000 | 10 to 50 GPU hours |
| A new capability | 50,000+ | considerable, often not sensible |
Measuring catastrophic forgetting
A model retrained on a narrow task loses general capability. That is not a side effect but the direct consequence of shifting all parameters towards the new task.
- Run the same general evaluation set before and after training.
- A drop of more than 3 to 5 percentage points is a warning sign.
- Remedies: smaller learning rate, fewer epochs, LoRA rather than full training, mixing in general data.
A proven rule of thumb is to blend 5 to 20 percent general instruction data into the training set. That costs little and keeps base capabilities largely stable.
The settings
| Parameter | Usual range | Note |
|---|---|---|
| Learning rate | 1e-5 to 5e-5 full, 1e-4 to 3e-4 with LoRA | The most important dial |
| Epochs | 1 to 3 | More almost always leads to memorisation |
| Effective batch size | 32 to 128 | Reachable via gradient accumulation |
| Maximum length | as short as possible | Determines the memory need |
| Warmup | 3 to 10 percent of steps | Prevents early instability |
Computed: memory and steps
Memory decides whether a fine-tuning run fits on the hardware you have at all.
For a model with 7 billion parameters in bfloat16 with Adam:
7e9 · (2 + 2 + 8) = 8.4 · 10¹⁰ bytes, so around 84 GB before activations are
counted. That fits on no single ordinary card.
With LoRA only the adapter matrices are trained. At rank 16 on the four
projections of a model with 32 layers and width 4096 that is
32 · 4 · 2 · 16 · 4096 ≈ 1.7 · 10⁷ parameters, or 0.24 percent of the model.
The optimiser state shrinks accordingly: 1.7e7 · 12 ≈ 200 MB instead of 84 GB.
The base model stays frozen and can be loaded quantised. See
LoRA and parameter-efficient training.
The number of steps follows from the dataset:
steps = examples · epochs / effective batch size. At 2,000 examples, 2 epochs
and an effective batch size of 32 that is 125 steps. That is few, and it is
exactly why the learning rate decides more here than compute time does.
Legal points
- Training data containing personal data needs its own legal basis. The purpose of the original collection rarely covers training.
- A model can reproduce training data verbatim. For personal data that is a transmission risk; for protected texts a copyright one.
- An erasure request under Art. 17 GDPR cannot readily be honoured on a trained model. Which is why personal data belongs removed before training, not after.
- Fine-tuning a model and passing it on can make you a provider under the AI Act. See Duties by role.
Related courses and sources
BERT
Pre-training on masked text, then fine-tuning per task. The pattern that shaped language processing before the large models.
For understanding what shaped language processing before the large models.
Hugging Face NLP course
Tokenisation, transformers, fine-tuning and deployment, with code throughout. Assumes Python, and in exchange you end up working with real models.
For development with language models once it has to go beyond calling an interface.
LoRA
Adaptation through a few additional parameters instead of full training. The reason fine-tuning is affordable today.
For anyone fine-tuning on a budget; the basis of affordable adaptation.
Practical Deep Learning for Coders
Starts with a working model in the first hour and supplies the theory afterwards. The shortest route from basic Python to a model you trained yourself.
For impatient readers who know Python: your first model runs within the first hour.