Holy shit, 2.3 TB of memory per chip! That's not your typical HBM stack - we're talking about a completely different memory architecture here. For context, current high-end GPUs max out around 192GB HBM3. This is likely either:
1) Next-gen HBM stacking with insane vertical integration (think 24+ layers instead of the current 8-12)
2) A hybrid DRAM-flash architecture that trades some latency for massive capacity
3) CXL-attached memory modules that technically count as "per chip" but aren't on-die
The real question: what's the bandwidth? You can have terabytes of memory, but if you're bottlenecked at PCIe speeds, it's useless for training large models. If this maintains anywhere near HBM3's 3+ TB/s bandwidth at 2.3TB capacity, we're looking at a total game-changer for inference serving and mixture-of-experts models that need to keep massive parameter sets in fast memory.
This could finally make 405B+ parameter models actually practical for real-time inference without model sharding across multiple nodes. The memory wall just got a lot taller.
1) Next-gen HBM stacking with insane vertical integration (think 24+ layers instead of the current 8-12)
2) A hybrid DRAM-flash architecture that trades some latency for massive capacity
3) CXL-attached memory modules that technically count as "per chip" but aren't on-die
The real question: what's the bandwidth? You can have terabytes of memory, but if you're bottlenecked at PCIe speeds, it's useless for training large models. If this maintains anywhere near HBM3's 3+ TB/s bandwidth at 2.3TB capacity, we're looking at a total game-changer for inference serving and mixture-of-experts models that need to keep massive parameter sets in fast memory.
This could finally make 405B+ parameter models actually practical for real-time inference without model sharding across multiple nodes. The memory wall just got a lot taller.