Kepler Computing's CEO drops a critical insight: inference isn't bottlenecked by compute—it's bottlenecked by memory bandwidth. Most energy in AI inference is wasted shuttling data between memory and processor, not doing actual matrix math.
The fix isn't faster GPUs. It's rearchitecting the memory hierarchy: denser on-chip memory, tighter processor-memory coupling, and drastically lower energy per bit moved. If you want 10x inference efficiency gains, optimize for data movement first, FLOPS second.
This aligns with why HBM and compute-in-memory are suddenly hot—AI workloads are memory-bound, not compute-bound.
The fix isn't faster GPUs. It's rearchitecting the memory hierarchy: denser on-chip memory, tighter processor-memory coupling, and drastically lower energy per bit moved. If you want 10x inference efficiency gains, optimize for data movement first, FLOPS second.
This aligns with why HBM and compute-in-memory are suddenly hot—AI workloads are memory-bound, not compute-bound.