Sinhala costs 3.5× more tokens than English, and nobody's counting

There is a rule of thumb that has been copied into approximately every LLM cost calculator on the internet: four characters per token. It comes from OpenAI's own early documentation, it is roughly right for English prose, and it is the number that gets baked into budget spreadsheets and rate limiters and "will this fit in the context window" checks.

It is also wrong for most of the world's writing systems, and I only noticed because I was building a token counter and decided to actually measure.

What the measurements look like

I ran samples through o200k_base — the tokenizer GPT models use, which is public and can be run entirely locally. Same tokenizer, same method, different scripts:

Text type	Characters per token
English prose	4.40
Source code	2.77
Sinhala prose	1.24

English lands almost exactly on the folk wisdom. Code is meaningfully denser — punctuation and indentation fragment into their own tokens, which is why a 500-line file eats context faster than a 500-line essay.

And Sinhala sits at 1.24. Just over one character per token.

That is 3.5× more tokens for the same number of characters than English. Not 10% worse. Three and a half times.

Your numbers will drift depending on corpus and tokenizer version. The ratio between scripts is the stable finding, not the third decimal place.

Why this happens

Tokenizers are trained on a corpus. Byte-pair encoding builds its vocabulary greedily: sequences that appear often become single tokens, sequences that appear rarely get decomposed into smaller pieces, and anything genuinely unusual falls all the way back to individual UTF-8 bytes.

English is enormously over-represented in the training corpora these tokenizers were built from. So common English words — and even common English word fragments — get their own dedicated token. understanding might be two tokens. the is one.

Sinhala is not over-represented. It is barely represented. So Sinhala text falls back toward byte-level encoding, and here the second problem compounds the first: Sinhala is written in an abugida with combining vowel signs, and a single perceived character routinely occupies three or more bytes in UTF-8.

Rare script plus multi-byte codepoints plus byte-level fallback equals roughly one token per character. Sometimes worse.

The part that actually matters

This is not a trivia fact. It has three concrete consequences, and I have hit all three.

Cost. If you bill per token — and you do — then serving Sinhala users costs you three and a half times more per character of text than serving English users. A support-chat product with identical usage patterns in both languages has wildly different unit economics depending on which market it grows in. Nobody models this, because everyone uses the four-characters-per-token assumption for both.

Context windows. A 200k-token context window is not a 200k-token context window. It is roughly 880,000 characters of English, or roughly 248,000 characters of Sinhala. If you built a document-ingestion feature and tested it in English, your "this comfortably fits" assumption breaks the first time a Sinhala document arrives at the same page count. It does not degrade gracefully; it truncates.

Quality. This one is less obvious and more annoying. When a tokenizer shreds a language into near-byte-level fragments, the model has less structure to work with. The tokens carry less meaning individually. Empirically, output quality in heavily-fragmented scripts tends to be worse than in scripts the tokenizer handles well — and you cannot fix that with a better prompt.

What to do about it

Measure your actual languages. Do not trust a global ratio. Take a representative sample of the text your product actually processes, run it through the real tokenizer, and divide. It takes ten minutes. For GPT models, gpt-tokenizer runs locally with no API call and no key.

Split by script before you estimate. This is the approach I ended up taking: partition the text by Unicode block, apply a separately-calibrated ratio to each partition, and sum. A mixed English-and-Sinhala message is common in Sri Lanka, and a single blended ratio is wrong for both halves of it.

Budget per-language, not per-user. If your pricing model assumes uniform token cost per message, and your user base is multilingual, your margins vary by market in ways your dashboard is not showing you.

Push back on the default. Most token counters are still shipping the flat four-characters rule. If you are building one, calibrate. If you are using one, check what it assumes before you trust its number.

The broader point

Every layer of the stack has defaults, and defaults encode the assumptions of whoever wrote them first. The four-characters-per-token rule is not a conspiracy; it is just a reasonable English-language approximation that escaped its context and became a universal constant.

Most of these assumptions are invisible until you work outside the environment they were written in. Then they are extremely visible, and usually expensive.

I built TokenLens partly to stop guessing about this one.