Tokenizer Fertility: Why AI Models Charge Kashmiri a ‘Token Tax’

Tokenizer fertility is the average number of subword tokens a model's tokenizer needs to represent one word. Tokenizers built mostly for English break Kashmiri, written in Perso-Arabic script, into many more pieces per word than English, so the same message costs more tokens, uses more of the context window and runs slower. Our ICRTC-2026 paper reported 5.89 tokens per Kashmiri word for Mistral-7B's tokenizer against 1.45 for English, about 4.06 times as many.

This post explains what a token is, why Kashmiri fragments, what the extra tokens cost, and what researchers have done about it. It is a worked companion to the fertility finding summarised on our research page and facts page.

What a token is

Language models do not read letters or words directly. A tokenizer first cuts text into units from a fixed vocabulary, and each unit becomes an integer ID the model processes. Most current models use subword tokenizers, such as byte-pair encoding (BPE) or SentencePiece. Common strings get a single token of their own; rarer strings are spelled out from smaller pieces.

The vocabulary is learned from training text. Sequences that appear often in that text are merged into longer tokens. Sequences that appear rarely stay as short pieces, sometimes a single character or a single byte.

What fertility measures

Rust et al. (2021), in How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models, define subword fertility as “the average number of subwords produced per tokenized word”, following Ács (2019). A fertility of 1 means every word in the text is a single token. Higher values mean the tokenizer is splitting words into more pieces.

Rust et al. found that multilingual BERT had much higher fertility than dedicated monolingual tokenizers for Arabic, Finnish, Korean, Russian and Turkish, and that lower fertility correlated with better downstream results. Replacing the shared tokenizer with a language-specific one improved performance even when the pretraining data was held constant. Fertility is a simple number, but it tracks something that matters.

Two cautions apply. Fertility depends on how “word” is counted (here, text between spaces), and on the text itself: Kashmiri written with full vowel marks produces different counts from the same words written without them.

Why Perso-Arabic Kashmiri fragments

Kashmiri in Perso-Arabic script uses letters and vowel marks that are rare in other Arabic-script text, such as ٲ (Arabic letter alef with wavy hamza above, U+0672) and the vowel sign ٛ (U+065B). Kashmiri text is a tiny share of the web, so a general-purpose tokenizer sees little of it when its vocabulary is learned. Few Kashmiri sequences earn their own merged tokens.

When a character is not in the vocabulary at all, some tokenizers fall back to raw bytes. Mistral-7B's tokenizer is a BPE model with a 32,000-entry vocabulary and byte fallback switched on (we checked its published tokenizer.json). Each Perso-Arabic character takes two bytes in UTF-8, so an unknown character becomes two byte tokens, and a word becomes a near character-by-character sequence. The NLLB team describe the same effect for under-represented languages: the tokenizer drifts towards “a near character-based model for those languages” (NLLB Team, 2022, §8.1.1).

A worked example

To make this concrete, we took one sentence from the introduction of the Kashmiri Wikipedia article on the Kashmiri language and counted tokens with four tokenizers on 8 October 2026:

کٲشِر تہِ چھُ ہِنٛدُستان چہِ 22 ہن درج شُدٕ زبانن مَنٛز شٲمِل۔

Our English rendering: “Kashmiri is also included among India's 22 scheduled languages.”

The Kashmiri sentence has 12 space-separated words; our English rendering has 9. Token counts exclude special tokens such as beginning-of-sentence markers.

In Mistral's output the first word, کٲشِر, becomes seven tokens: a word-boundary marker, ک, two raw bytes for ٲ, then ش, the vowel mark and ر. The word for India, ہِنٛدُستان, becomes twelve. NLLB's tokenizer keeps pieces such as ستان and زبان whole.

This is a single, fully vowel-marked sentence, so treat it as an illustration, not a measurement. The paper's figures come from a much larger sample, described below.

What it costs

Petrov et al. (2023), in Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023), found that the same text translated into different languages can differ in tokenized length by up to 15 times, and that even tokenizers designed to be multilingual, and character- and byte-level models, show more than four-fold differences for some language pairs. The extra tokens show up in four places:

• Price. Commercial APIs charge per token. Ahia et al. (2023), in Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, studied 22 languages and found that speakers of many supported languages are overcharged while getting poorer results, often in regions where the APIs are already less affordable.
• Context window. A model's context limit is set in tokens, so a language with three to four times the fertility fits roughly a third to a quarter as much text into the same window.
• Speed. Generation produces one token at a time, so more tokens per word means longer waits for the same output.
• Accuracy. Lundin et al. (2025), in The Token Tax: Systematic Bias in Multilingual Tokenization, tested 10 language models on AfriMMLU (16 African languages) and report that fertility reliably predicts accuracy: higher fertility, lower scores. They also state that doubling the number of tokens quadruples training cost and time, and they call this disadvantage a “token tax”.

What the ICRTC-2026 paper reported

Our paper, Anatomy of Decoder-Only LLM Failure in Low-Resource Machine Translation: Tokenizer Fertility, Data Thresholds, and Decoding Strategies for Kashmiri (Best Paper, ICRTC-2026, Springer), compared a QLoRA-fine-tuned Mistral-7B with NLLB-200 and IndicTrans2 on Kashmiri→English translation. As reported in the paper:

• Mistral's BPE tokenizer produced 5.89 tokens per Kashmiri word against 1.45 per English word, a 4.06× gap.
• NLLB-200's SentencePiece tokenizer came close to parity: 2.50 against 1.40.
• The paper identified this fragmentation as the upstream cause of the fine-tuned LLM's failures: the context fills with source-encoding overhead, and morphemes are split arbitrarily across subword boundaries. Fine-tuned Mistral-7B scored 13.89 BLEU on the 500-sentence test set, against 31.57 for NLLB-200-600M, a model less than a tenth of its size.

These results are as published. Follow-up work in 2026 is re-testing them, and the details will appear in a follow-up paper. The research page has the full summary.

What helps

Choose a model with a multilingual tokenizer

The NLLB team trained one SentencePiece model for more than 200 languages with a 256,000-token vocabulary. To stop high-resource languages dominating it, they sampled training sentences with temperature sampling (T = 5), which upsamples low-resource languages, and they checked the tokenization rate per language (NLLB Team, 2022, §8.1.1). Kashmiri is one of the NLLB-200 languages, which is part of why a 600M-parameter NLLB model handled it better than a 7B general LLM in our paper. Our translator page covers the models we use for Kashmiri today.

Extend the vocabulary

Another route is to add new tokens for the target language to an existing model and continue training it. Cui, Yang and Yao (2023) added about 20,000 Chinese tokens to LLaMA's vocabulary in Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca, which improved encoding efficiency and Chinese performance. The new token embeddings start untrained, so this needs a substantial amount of target-language text to pay off, which is scarce for Kashmiri.

Use a dedicated tokenizer

Rust et al. (2021) showed that swapping in a language-specific tokenizer improved results even with the same pretraining data. A custom Kashmiri SentencePiece tokenizer is listed as future work in our paper.

Byte-level models

Byte-level models such as ByT5 (Xue et al., 2022, ByT5: Towards a token-free future with pre-trained byte-to-byte models, TACL) skip the learned vocabulary and read raw UTF-8 bytes, so they can process text in any language and are more robust to noise and spelling variation. They do not remove the length gap: Perso-Arabic characters still take two bytes each, and Petrov et al. found more than four-fold differences in encoding length for some language pairs even at the byte level.

For the wider picture of why languages such as Kashmiri are underserved, see our post on low-resource language NLP.

Sources

• Rust, P., Pfeiffer, J., Vulić, I., Ruder, S. and Gurevych, I. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. ACL-IJCNLP 2021.
• Petrov, A., La Malfa, E., Torr, P. H. S. and Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023.
• Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A. and Tsvetkov, Y. (2023). Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models.
• Lundin, J. M., Zhang, A., Karim, N., Louzan, H., Wei, V., Adelani, D. and Carroll, C. (2025). The Token Tax: Systematic Bias in Multilingual Tokenization.
• NLLB Team et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation.
• Cui, Y., Yang, Z. and Yao, X. (2023). Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca.
• Xue, L. et al. (2022). ByT5: Towards a token-free future with pre-trained byte-to-byte models. TACL.
• Example sentence: Kashmiri Wikipedia, article on the Kashmiri language.