AI chip startup Groq, which was acquired by NVIDIA, is reportedly planning to overhaul its chip memory design. Korean media outlet ZDNet reported on October 20 that Groq is evaluating the use of 3D packaging technology in its next-generation chips, replacing SRAM—the core of its original design—with DRAM, with the aim of significantly increasing the memory capacity of each chip.

A new direction for Groq’s next-generation chip design! Plans to replace SRAM with DRAM

ZDNet, citing two anonymous semiconductor industry sources, reported that Groq is reviewing the use of 3D packaging technology in its next-generation chips, replacing SRAM in the current design with DRAM. One industry source said Groq plans to use 3D packaging in its new chips, stacking DRAM instead of continuing to use SRAM.

3D packaging vertically stacks multiple semiconductor dies so they function like a single chip. Reports say that compared with traditional 2D planar designs, this approach can reduce the footprint and may also improve power efficiency.

Groq was founded in 2016 by Jonathan Ross, one of the designers of Google’s TPU, and was acquired by Nvidia last year. Previous reports that its “Groq 3 LPX” had entered mass production suggest that Groq’s product line is continuing to advance within Nvidia.

What Is 3DS? How It Differs from HBM in Capacity

Reports say that Groq is reportedly using 3DS (3D Stacked Memory), which vertically connects DRAM dies using through-silicon vias (TSVs) to create a high-capacity package. Like high-bandwidth memory (HBM), 3DS uses TSVs, but the two technologies have different design goals. HBM focuses on data transfer rates, while 3DS retains the existing DRAM interface and focuses on expanding capacity.

Groq is not the only company advancing memory-stacking technology. As Chain News previously reported, SK hynix has begun commercializing 3D-stacked DRAM-on-Logic. The technology stacks DRAM directly on top of a logic chip, shortening physical distances and increasing data transfer channels to overcome the bandwidth and power-consumption limitations of current PoP packaging. It primarily targets AI in mobile devices.

The difference lies in the architecture. At present, all that is known about Groq 3DS is that it stacks DRAM to expand capacity; it has not been specified whether the DRAM is stacked on top of the compute die.

(Is 3D-stacked DRAM a prerequisite for bringing AI to devices? SK hynix reportedly teams up with Apple to pursue next-generation memory)

Why DRAM? A capacity trade-off tailored to AI inference

ZDNet reports that Groq’s previous designs centered on SRAM, which is fast but offers limited capacity. DRAM is comparatively slower, but better suited to providing large capacities. The report interprets Groq’s move as an effort to overcome the capacity limitations of its existing design. Another industry insider observes that the market as a whole is moving toward larger chips, prompting Groq to choose DRAM, which offers an advantage in capacity.

From the perspective of AI inference, this trade-off makes considerable sense. As LLM parameters and context lengths continue to grow, memory capacity requirements are also rising, and relying solely on SRAM can be limiting.

However, SRAM’s advantages lie in its speed and latency, and it remains unclear whether these characteristics can be maintained after switching to DRAM. Further clarification from Groq or Nvidia is needed to determine whether this design shift is real and what its practical impact will be.

(Memory hierarchy reshuffling in the AI inference era: A guide to the differences between HBF, SSD POD, and SRAM)

This article, “Nvidia Moves into Next-Generation Memory Chips! Groq Reportedly Adopts 3D-Stacked DRAM to Meet AI Inference Demand,” originally appeared on Chain News ABMedia.