What it means
Models do not process letters or words; they process tokens. A tokenizer splits text into pieces drawn from a fixed vocabulary — common words are typically one token, rarer words split into several, and punctuation and spaces count too. For ordinary English, a useful rule of thumb is that a token averages around three-quarters of a word.
Tokenization explains several odd behaviors. Models have historically struggled to count letters in a word or reverse a string, because they never see the letters — they see an opaque chunk. Unusual names, technical strings and non-Latin scripts fragment into many more tokens, which is why they consume more context and cost more.
Everything downstream is measured in tokens: context window size, API pricing, and rate limits.
Why it matters
Tokens are the billing unit and the capacity unit, so estimating them is how you estimate both cost and whether something fits. The uneven tokenization across languages also has an equity dimension — the same message can cost several times more in some languages than in English.
In practice
Estimate with the three-quarters-of-a-word rule for English prose, then verify with a real tokenizer for anything cost-sensitive. Code, JSON, and non-English text all tokenize much less efficiently than the rule suggests.