When is data parallelism enough?

As long as the model plus optimiser states fit one card. Beyond that you need methods that also shard the parameters.

Why does it not scale linearly with card count?

Because gradients are exchanged between all cards after every step. That exchange grows with card count and eventually becomes the bottleneck.

What are FSDP and ZeRO?

Methods that shard parameters, gradients and optimiser states across cards and gather them only when needed. Memory per card then falls almost linearly.

Distributed training

The idea

Training a large model on many cards is like a building project with many teams. The work can be split three ways: every team builds its own house to the same plan, or all build one house and share the walls, or each team takes one storey.

Each split has different coordination costs, and coordination is the bottleneck.

The three axes

Axis	What is split	Communication
Data parallelism	Different examples per card	Gradients after every step
Tensor parallelism	One layer across several cards	After every layer
Pipeline parallelism	Different layers per card	Activations between stages

Which split when

01
Everything fits one card
Plain data parallelism. Simple, robust, scales to the communication limit.
02
Optimiser states are too large
ZeRO stages 1 to 3 or FSDP. States are sharded and gathered on demand.
03
A single layer no longer fits
Tensor parallelism, strictly within one node with a fast interconnect. Across node boundaries it destroys throughput.
04
Very many layers
Pipeline parallelism with micro-batches to keep idle time small.

Communication cost

All-reduce of the gradients

volume per card = 2 · N · b · (p − 1)/p ≈ 2 · N · b

With ring all-reduce each card sends roughly twice the parameter volume, independent of card count.

N: number of parameters
b: bytes per gradient
p: number of cards

For a 7-billion-parameter model in bfloat16 that is around 28 GB per card per step. Over a 400 GB/s link the exchange takes about 70 milliseconds. If a training step itself takes 200 milliseconds, 26 percent of the time is lost to communication, unless it is overlapped with the backward pass, which modern libraries do.

The pipeline bubble

Bubble efficiency

idle share = (s − 1) / (m + s − 1)

The share of lost time falls the more micro-batches are pushed through the pipeline.

s: number of pipeline stages
m: number of micro-batches per step

With 8 stages and 8 micro-batches, 7/15 = 46.7 percent is lost. With 64 micro-batches only 7/71 = 9.9 percent. That is why pipeline parallelism always runs with a large number of small micro-batches, and why it does not work with small overall batches.

Memory per card under ZeRO

Stage	What is sharded	Memory per card with p cards
0	nothing	`16 · N` bytes
1	optimiser states	`4 · N + 12 · N / p`
2	plus gradients	`2 · N + 14 · N / p`
3	plus parameters	`16 · N / p`

At N = 7e9 and 8 cards that is 112 GB per card unsharded and 14 GB per card at stage 3: the difference between impossible and feasible. The price is extra communication, because parameters are gathered before each layer and discarded after.

Reproducibility

A distributed run is not bit-reproducible even with a fixed seed, because the order of gradient summation depends on network timing. For evidence purposes that means: the run is not what gets attested, the result is. Model checksum, dataset version, configuration and metrics belong in the log. See Logging.

The idea#

The three axes#

Which split when#

Communication cost#

The pipeline bubble#

Memory per card under ZeRO#

Reproducibility#