Want to run 770B Hy4 on a 128GB Spark? Someone has already found a way—but the speed still leaves a lot to be desired.
Hy4 is open source, and with only 49B active parameters, it’s easy to think a 128GB Spark can handle it. But it has 770B parameters in total, requiring over 700GB even at FP8 precision. With the usual approach, all those weights have to fit in memory.
What if you tried a different approach? Load only the experts needed at the time from disk.
Someone has actually done it. Using the original DeepSeek weights on a single Spark, with experts loaded straight from disk, they got 4.4 tok/s.
It works, but that speed is still a long way from practical everyday use. Once it gets to around 40 tok/s, it might be worth seriously considering running models this large on a single machine.
Hy4 is open source, and with only 49B active parameters, it’s easy to think a 128GB Spark can handle it. But it has 770B parameters in total, requiring over 700GB even at FP8 precision. With the usual approach, all those weights have to fit in memory.
What if you tried a different approach? Load only the experts needed at the time from disk.
Someone has actually done it. Using the original DeepSeek weights on a single Spark, with experts loaded straight from disk, they got 4.4 tok/s.
It works, but that speed is still a long way from practical everyday use. Once it gets to around 40 tok/s, it might be worth seriously considering running models this large on a single machine.
