Whenever an AI company releases a new model, one of the most common questions in the market is: “How many parameters does it have?”
But Tsinghua University professor and founder of Z.ai (Zhipu AI), Jie Tang, believes that discussing parameter count alone has already lost its meaning.
Looking back on the Scaling Law evolution from GPT-3 and Chinchilla to mixture-of-experts (MoE) models, he points out that model capability depends not only on parameters, but also on training data size, where the compute budget is spent across different stages, and who will ultimately use the model and under what conditions.
More importantly, as AI moves from a “train once, test once” paradigm to tens of billions of inferences every day, the optimal solution for Scaling Law has kept changing.
Jingwei Tang believes that once model parameters reach the threshold to “contain world knowledge,” continuing to stack up total parameters may not be the best way to improve capabilities. The next stage may be more worth expanding in terms of the effective depth of single-turn inference and post-training.
And Zhipu’s latest GLM-5.3 is precisely a control experiment conducted to test this hypothesis.
From GPT-3 to Chinchilla: the AI industry once collectively went through the “billion-parameter detour”
Jingwei Tang first reviews the development history of scaling laws.
In 2020, the OpenAI research team led by Kaplan et al. published Scaling Laws for Neural Language Models. Their results were once interpreted as: as compute increases, model parameters should grow faster than training data, with expansion speeds of roughly 2.7:1 for the two.
This scaling law profoundly influenced the direction of large-model development at the time. Models such as GPT-3, DeepMind Gopher, and Microsoft MT-NLG all moved toward larger parameter scales, and the idea of “the bigger the model, the better” gradually became the mainstream view across the industry.
However, after DeepMind published the Chinchilla paper in 2022, this understanding was largely corrected.
Researchers such as Hoffmann re-ran experiments with about 400 models and found that if you pursue the best results under a fixed training compute budget, model parameters and training data should grow more evenly, with a classic rule of thumb of about 20 training tokens per parameter.
Those large models back then weren’t necessarily suffering from too few parameters; instead, they may have been “made too big while not feeding enough data.”
Jingwei Tang points out that the estimation error in the Kaplan scaling law grows as the compute scale increases. Therefore, the largest model in that generation might also be the one most severely deviating from the optimal allocation point.
He describes it this way: looking back, the billion-parameter race was a detour the whole field took together—only to turn back together at the end.
Chinchilla is not the endpoint either
However, what Chinchilla mainly solved was still the “training compute optimization” problem.
Its basic assumption leans toward evaluating a model after one round of training, but today’s commercial AI models may be called tens of billions of times every day. Once the inference cost after deployment is incorporated into considerations across the model’s entire lifecycle, the best model configuration changes again.
Jingwei Tang says that if inference cost is included in the optimization objective, a reasonable strategy might instead be: make the model smaller, but train it longer. That is to intentionally do “over-training.”
He used Meta’s Llama-2-7B and Google’s Gemma-2-9B as examples: their training volumes reach about 290 tokens per parameter (TPP) and 889 TPP respectively—far higher than Chinchilla’s classic 20 TPP.
This isn’t simply wasting more training compute. Instead, in situations where the model will be deployed widely and perform repeated inference, it invests more training cost upfront in exchange for a smaller, cheaper inference model.
MoE changes the scaling law again
After mixture-of-experts (MoE) became widespread, it became even harder to gauge capability just by “how many parameters the model has.”
Jingwei Tang believes that MoE must at least distinguish between two concepts:
Total parameters largely determine how much the model can “fit,” such as knowledge, facts, and vast long-tail information. Activated parameters and effective depth are more closely related to how complex a computation a model can carry out from start to finish in a single inference—and whether it can maintain a relatively long causal reasoning chain.
Therefore, directly applying the Dense Model’s “20 tokens per parameter” rule to an MoE model may itself be problematic.
Jingwei Tang further cited research by Roberts and others in 2025, pointing out that the optimal tokens-per-parameter is not a fixed constant; instead, it depends on the task. Memory-based tasks tend to favor increasing parameters, while reasoning-based tasks benefit more from increasing data.
Subsequent MoE research even observed that under a fixed TPP, simply increasing the model’s total parameters may actually reduce reasoning ability. By contrast, increasing the number of Experts that actively participate in computation can more reliably improve reasoning performance.
This also makes the question of “How many billions of parameters does this model have?” increasingly difficult to answer on its own in terms of how capable an AI model really is.
Finding loopholes isn’t data retrieval. What’s truly difficult is completing a 20-step chain of reasoning.
Jingwei Tang specifically used cybersecurity vulnerabilities as an example to explain the difference between “knowing a lot” and “reasoning very deeply.”
If a model is to discover a new software vulnerability, the real capability it needs is not memorizing more CVE vulnerability data, but whether it can work through an inference chain that may be as long as 20 steps to the end—while still maintaining correct context and causal relationships throughout the process.
How much world knowledge the model memorizes and how deeply it can think through a problem may be two different kinds of scaling problems.
Jingwei Tang points out that the total number of parameters may only be very important up to a certain threshold. The model first must be “big enough to contain the world.” Once it crosses that threshold, new capabilities may come more from other scaling dimensions, including the effective depth of a single forward pass and post-training.
GLM-5.3 runs a control experiment: no increase in parameters, but expand RL and long-horizon task training
He said that GLM-5.3 can be seen as a control experiment by Zhipu to test this scaling hypothesis.
GLM-5.3 and GLM-5.2 use the same Base Model, the same architecture, and even keep the same total parameters and startup parameters. This time, the team did not choose to keep scaling up the model; instead, it spent one month expanding Long-horizon Environments and strengthening learning (RL).
In other words, this experiment intentionally controlled the model’s size, and only turned one knob: “Post-training Scaling.”
According to Jingwei Tang, the resulting capability improvements are not merely marginal—they are quite significant.
This result also led him to a more important conclusion: scaling hasn’t failed. The AI industry may just need to re-understand what “scaling” is actually expanding.
Jingwei Tang does not believe this means model parameters are no longer important, nor has he declared that Pre-training Scaling has reached its limit.
On the contrary, there is still room for expansion in the future, including Base Model Size, Pre-training Data, and the compute invested in each Forward Pass—Zhipu also expects to return to these directions.
The article: “The bigger the parameters, the not necessarily the stronger!” Zhipu AI founder Jingwei Tang discusses scaling laws: the key is reasoning depth and post-training. First appeared in .