Why throwing more GPUs at training doesn't always = faster training?
The pipeline bubble problem.
When you split a model across machines (pipeline parallelism), each stage sits idle waiting for the previous stage's output. That dead time where hardware is literally doing nothing? That's your bubble.
Basically: Machine 2 can't start until Machine 1 finishes its forward pass. Machine 3 waits on Machine 2. The sequential dependency creates gaps where expensive compute just burns money doing nothing.
This is why naive model parallelism often has terrible GPU utilization - you're paying for 8 GPUs but only 2-3 are actually working at any given moment. The rest are waiting in the pipeline queue.
Solutions involve micro-batching (splitting batches into smaller chunks to keep pipes full) or interleaving forward/backward passes, but the fundamental constraint remains: pipeline stages create inherent serialization that kills your scaling efficiency.
The pipeline bubble problem.
When you split a model across machines (pipeline parallelism), each stage sits idle waiting for the previous stage's output. That dead time where hardware is literally doing nothing? That's your bubble.
Basically: Machine 2 can't start until Machine 1 finishes its forward pass. Machine 3 waits on Machine 2. The sequential dependency creates gaps where expensive compute just burns money doing nothing.
This is why naive model parallelism often has terrible GPU utilization - you're paying for 8 GPUs but only 2-3 are actually working at any given moment. The rest are waiting in the pipeline queue.
Solutions involve micro-batching (splitting batches into smaller chunks to keep pipes full) or interleaving forward/backward passes, but the fundamental constraint remains: pipeline stages create inherent serialization that kills your scaling efficiency.