Data parallelism = copy entire model to each GPU, split dataset across them. Works great until you hit 70B+ parameter models that literally won't fit in consumer VRAM (24GB won't cut it).
Model parallelism = split the model itself across GPUs. Fixes the memory bottleneck but now you're constantly shuffling tensors between devices. At scale, that inter-GPU communication becomes your new performance killer.
The tradeoff: either you can't fit the model, or you can fit it but spend half your time waiting on PCIe/NVLink bandwidth. This is why hybrid approaches (pipeline + tensor parallelism) exist.
Model parallelism = split the model itself across GPUs. Fixes the memory bottleneck but now you're constantly shuffling tensors between devices. At scale, that inter-GPU communication becomes your new performance killer.
The tradeoff: either you can't fit the model, or you can fit it but spend half your time waiting on PCIe/NVLink bandwidth. This is why hybrid approaches (pipeline + tensor parallelism) exist.