Mistral AI's Sagar Vaze just dropped some serious efficiency math on pretraining compute. Mistral Large 4 uses ~10% of the FLOPs compared to traditional dense models but still crushes benchmarks in finance, legal, and agentic coding.
The trick? Sparse MoE architecture. GPT-3 was a dense 175B param model—every single parameter fired on every token. Mistral Large 4 has 1 trillion total params but only activates ~50B per token (less than a third of GPT-3's active count).
Since training FLOPs scale with active params per token, not total params, you can pack way more capacity without proportional compute cost. This basically breaks the old "bigger model = more FLOPs" rule.
Vaze also points out that FLOP benchmarks are moving targets—pretraining efficiency keeps improving, so yesterday's "minimum viable compute" is irrelevant tomorrow. The real game is params-per-token efficiency, not raw model size.
The trick? Sparse MoE architecture. GPT-3 was a dense 175B param model—every single parameter fired on every token. Mistral Large 4 has 1 trillion total params but only activates ~50B per token (less than a third of GPT-3's active count).
Since training FLOPs scale with active params per token, not total params, you can pack way more capacity without proportional compute cost. This basically breaks the old "bigger model = more FLOPs" rule.
Vaze also points out that FLOP benchmarks are moving targets—pretraining efficiency keeps improving, so yesterday's "minimum viable compute" is irrelevant tomorrow. The real game is params-per-token efficiency, not raw model size.