Training large LLMs = memory vs communication tradeoff
Data parallelism: Copy full model to every node → low communication overhead but memory-heavy per node
Model parallelism: Split model across nodes → saves memory but communication bottleneck kills you
Real world: You need hybrid strategies (pipeline parallelism + tensor parallelism + ZeRO optimization) to scale beyond toy models. Pure data or pure model parallelism doesn't cut it for 100B+ param models.
Data parallelism: Copy full model to every node → low communication overhead but memory-heavy per node
Model parallelism: Split model across nodes → saves memory but communication bottleneck kills you
Real world: You need hybrid strategies (pipeline parallelism + tensor parallelism + ZeRO optimization) to scale beyond toy models. Pure data or pure model parallelism doesn't cut it for 100B+ param models.