Quick math for home-based distributed training:
A 7B param model = 7B gradients. At FP16 precision, that's 14GB per full gradient sync.
Raw upload times (no compression, no protocol overhead):
• 100 Mbps upload → ~19 min per sync
• 1 Gbps upload → ~2 min per sync
This is why gradient compression (TopK, quantization) and async updates are critical for federated learning over consumer connections. Even with gigabit fiber, you're bottlenecked by network I/O way before compute on most setups.
A 7B param model = 7B gradients. At FP16 precision, that's 14GB per full gradient sync.
Raw upload times (no compression, no protocol overhead):
• 100 Mbps upload → ~19 min per sync
• 1 Gbps upload → ~2 min per sync
This is why gradient compression (TopK, quantization) and async updates are critical for federated learning over consumer connections. Even with gigabit fiber, you're bottlenecked by network I/O way before compute on most setups.