GUIDE深度 · Deep dive

Tokenizer Differences: English, Chinese and Code

Tokenizers/vocabularies differ: English is often ~4 chars/token, Chinese 1-2 chars/token, and code/symbols/emoji fragment more. Character count is only a rough estimate; billing and hard limits require the model tokenizer.

更新 2026-10-31 · 公式可核验

Common differences

Chinese uses more tokens per character than English.

Code indentation/symbols add tokens.

Emoji/rare characters may use multiple tokens.

How to estimate

English at ~4 chars/token.

Chinese at 1.5-2 chars/token.

Verify critical cases with the official tokenizer.

FAQ

Why does the estimate differ from billing?

Tokenizer/tools/output/caching rules differ; billing is authoritative.

Is Chinese always more expensive?

Usually more per character; per information varies by model/content.

Sources

Related calculators

Token CounterPaste text to estimate tokens, characters, words and context usage for mixed English/Chinese/code; no API key.

More guides

Context Budget: Input, Output and System PromptRead guide