AI researcher Aran Komatsuzaki recently conducted a cross-language comparison of major models' tokenization tools. The results show that Claude, under Anthropic, performs poorly in handling Chinese, processing significantly less content than other models for the same cost, a phenomenon the community has dubbed the 'Chinese tax.'
Cross-language tokenization efficiency comparison
The researchers adopted the well-known AI paper (The Bitter Lesson) as a standard sample, translating it into nine languages and then inputting it into the tokenization tools of six models: OpenAI, Gemini, Qwen, DeepSeek, Kimi, and Claude.
The experiment uses 'the number of tokens for English original text on OpenAI's tokenization tool' as a baseline of 1x. Data shows significant performance disparities among models when processing the same Chinese content.
The token consumption of the Chinese language model Kimi is only 0.81 times the baseline, while Qwen is at 0.85 times. This means that in these models, the compression efficiency of Chinese even surpasses that of English.
OpenAI performs reasonably well, with Chinese token consumption at 1.15 times the baseline, reflecting some optimization of its tokenizer for mainstream foreign languages. The worst performer in Chinese processing is Claude, with consumption reaching 1.65 times the baseline, 50% more than OpenAI, and even double that of Kimi. In Hindi, Claude's consumption has even exceeded 3 times the baseline.
Follow-up on non-English token-inefficiency with more model-language pairs: – Chinese is cheaper than English on major Chinese models – Gemini and Qwen provide the least non-English tax – Anthropic has the highest tax by far; Kimi is next – Hindi is the worst-covered language here,… https://t.co/imeRUrzwjD pic.twitter.com/DEM84FmHoV
— Aran Komatsuzaki (@arankomatsuzaki) April 28, 2026
Why is the same Chinese more expensive on Claude?
The root of this gap lies in the training strategy of the 'tokenizer'. The tokenizer is the first barrier for AI models to understand text, responsible for breaking down text into token units.
Aran's conclusion notes: 'The larger the market, the more tokens you save.' The efficiency of the tokenizer highly depends on the proportion of languages in the training data. Domestic models like Kimi and Qwen have incorporated vast amounts of Chinese data in their training, allowing their tokenizers to efficiently 'package' Chinese words; in contrast, Anthropic's early data was heavily biased towards English, leading its tokenizer to adopt a 'fragmented cutting' strategy when processing Chinese, resulting in the same sentence being split into more chunks, thus increasing the total token count.
The real loss impact faced by users
Low tokenization efficiency is not just a technical indicator; it means multiple losses for users. Since API billing is based on tokens, Chinese users face nearly 65% higher development costs using Claude compared to other models. Moreover, the more tokens generated by the model, the longer the inference time, leading to noticeable delays for users.
On the flip side, although Claude is known for its ultra-long context window, due to low tokenization efficiency, the actual amount of information that can be accommodated in the same window capacity in Chinese scenarios is about 40% less than competitors, causing the model to hit its memory limit faster.
This cross-evaluation serves as a reminder to AI users that the level of language optimization will directly impact model costs. For heavy Chinese users, while pursuing the logical capabilities of the model, this 'Chinese tax' bill may be a reality to weigh before choosing Claude.
Source