GLM-5.3 clocks in at 320B params. Kimi K3 hits 2.8T. These aren't vaporware—devs are pulling these open weights right now.
The hard constraint everyone's hitting: single-node memory. 320B won't squeeze into one GPU's VRAM. At this scale, distributed training stops being a nice-to-have and becomes the only viable architecture. You're forced into multi-node setups, dealing with tensor parallelism, pipeline parallelism, or hybrid strategies just to load the model, let alone train or fine-tune it.
This is the new baseline: if you're working with frontier open models, your infra needs to be distributed from day one.
The hard constraint everyone's hitting: single-node memory. 320B won't squeeze into one GPU's VRAM. At this scale, distributed training stops being a nice-to-have and becomes the only viable architecture. You're forced into multi-node setups, dealing with tensor parallelism, pipeline parallelism, or hybrid strategies just to load the model, let alone train or fine-tune it.
This is the new baseline: if you're working with frontier open models, your infra needs to be distributed from day one.
