GUIDE深度 · Deep dive
Tokenizer Differences: English, Chinese and Code
Tokenizers/vocabularies differ: English is often ~4 chars/token, Chinese 1-2 chars/token, and code/symbols/emoji fragment more. Character count is only a rough estimate; billing and hard limits require the model tokenizer.
Common differences
Chinese uses more tokens per character than English.
Code indentation/symbols add tokens.
Emoji/rare characters may use multiple tokens.
How to estimate
English at ~4 chars/token.
Chinese at 1.5-2 chars/token.
Verify critical cases with the official tokenizer.
FAQ
Why does the estimate differ from billing?
Tokenizer/tools/output/caching rules differ; billing is authoritative.
Is Chinese always more expensive?
Usually more per character; per information varies by model/content.