Anatomy of Decoder-Only LLM Failure in Low-Resource Machine Translation

Getting a usable Kashmiri parallel corpus is harder than it sounds. We aggregate 124,102 pairs from two public repositories and apply a four-stage reproducible filtering pipeline.

• Deduplication and length — exact-match dedup, then a 10–500 character filter per side
• Script verification — Arabic Unicode character ratio ≥ 0.2, removing Latin-script misclassifications and code-switched fragments
• Semantic alignment — cosine similarity ≥ 0.4 between paraphrase-multilingual-MiniLM-L12-v2 embeddings, discarding content-misaligned rows
• Split — 95/5 into 117,896 training and 6,206 evaluation sentences

Translation quality across three systems

IndicTrans2’s numbers should be read as a performance ceiling rather than a fair zero-shot result: because the corpus draws on BPCC data, there is likely overlap with its pre-training set. BERTScore was computed only for the top-performing model, due to GPU memory constraints on the evaluation hardware.

The minimum data threshold

The most striking result appears when training data is reduced to the 30K subset. BLEU collapses from 13.89 to 0.46 — not a gradual decline, but near-total failure. A 4.2× data increase then produces a 29.9× BLEU gain, a shape that looks far more like a phase transition than a learning curve. We term this the cross-lingual transfer threshold.

Decoding is the highest-leverage intervention

An oracle experiment — truncating Mistral’s outputs to 1.2× the reference word count, changing nothing about the model — lifts BLEU from 13.89 to 21.74, closing 44% of the gap to NLLB-200. The correct translation is already in the output; the model simply does not know when to stop. This is a decoding problem, not a knowledge problem.

The oracle needs reference lengths, so it is not deployable. A source-constrained strategy — cap generation at 2× the source word count with a 1.3 repetition penalty — captures the same upside without them.

The baseline BLEU here (3.75) differs from Table 1 (13.89) because the inference configuration and the failure-heavy sentence subset differ; the relative gain is the meaningful number. The lesson: gentle penalties fail, hard length constraints work.

500 translations were manually reviewed and their errors traced to five recurring mechanisms.

• Over-generation (hallucination) — 66% of outputs run past 1.5× the reference word count. The average prediction/reference ratio is 2.2×, against 1.0× for NLLB-200. The model translates correctly, then keeps going.
• Tokenizer fragmentation — 5.89 tokens per Kashmiri word against 1.45 for English. The context window fills with source-encoding overhead, and morphemes get split arbitrarily across subword boundaries.
• Repetition loops — 47.2% of outputs show some degeneracy. The most characteristic is numeric sequence overflow: a plausible translation prefix, then counting forever. Pure word repetition accounts for 12.2%.
• Source copying — Kashmiri characters carried through into the English output, a sign the language transfer never fully triggered.
• Semantic drift — fluent English bearing little relation to the source. A grounding failure rather than a fluency one.

Contrastive analysis: where the gap actually lives

On the 15.8% of sentences where NLLB wins outright, its average BLEU is 47.7 against Mistral’s 7.6. These are not randomly distributed — they are the morphologically complex inputs that expose tokenizer fragmentation most sharply, supporting it as the upstream root cause.

What this means for other languages

All three mechanisms are structural rather than Kashmiri-specific. Arabic, Thai, Amharic, and Tigrinya show analogous fertility patterns. Three concrete lessons for practitioners on other non-Latin low-resource pairs: fix the tokenizer first — a joint SentencePiece vocabulary is the highest-return preprocessing investment; constrain generation length — even a hard cap on source word count recovers substantial quality without retraining; and treat 30K pairs as a floor, below which QLoRA fine-tuning should not be expected to produce useful translations.

Automated metrics rarely capture translation quality fully, and the disconnect widens for morphologically rich languages. To the best of our knowledge, this platform is the first tool built specifically to evaluate Kashmiri machine translation. Evaluators rate two anonymised systems on:

• Adequacy (1–5) — how much meaning from the source is preserved
• Fluency (1–5) — how natural and grammatical the translation reads
• Overall preference — which system produced the better translation

Design choices that protect annotation quality

• Word-level visual diffing — when two translations are close, differing words are highlighted automatically, lowering the cognitive cost of spotting small differences
• Full blinding and randomisation — model identities are hidden and side assignment is re-randomised for every sentence, so neither model nor screen position can bias a rating
• Anti-rushing telemetry — time spent per judgement is recorded, so implausibly fast responses can be filtered in post-processing
• Keyboard interface — number keys for the 1–5 scales and arrow keys for preference, letting frequent raters move quickly without losing accuracy

Early results

Across 123 blinded side-by-side comparisons, reviewers rated System A 3.88/5 for adequacy and 3.72/5 for fluency, and System B 3.63/5 and 3.41/5. On overall preference, native Kashmiri speakers chose System A 36.6% of the time, System B 32.5%, and a tie 30.9%. Inter-annotator agreement is tracked with Cohen’s weighted κ and pairwise preferences tested with the Wilcoxon signed-rank test. These figures are a baseline yardstick as more judgements accumulate.

• The BPCC web-crawled corpus likely retains residual noise the filtering pipeline did not remove, introducing some ambiguity into absolute metric scores.
• Only the Kashmiri→English direction was studied; the reverse direction may behave differently.
• Compute constraints (dual T4, Kaggle quota) prevented a thorough sweep over learning rates, epochs, and LoRA ranks.
• Human evaluation data is still being collected, so the study leans on automated metrics as a proxy for human judgement.
• IndicTrans2’s 52.88 BLEU is best treated as an upper bound, given likely test-set contamination from BPCC pre-training.
• The mechanistic claims are Kashmiri-specific; replication on Amharic, Uyghur, or another Perso-Arabic pair is needed before generalising.

Future work

A custom Kashmiri SentencePiece tokenizer; extended multi-epoch training beyond the 1,000-step cap; empirical comparison of beam search with coverage penalties against constrained decoding; bidirectional English→Kashmiri evaluation; and replication of the data threshold on other Perso-Arabic pairs.

Ayoub, F., and Tigga, N. P. (2026). Anatomy of Decoder-Only LLM Failure in Low-Resource Machine Translation: Tokenizer Fertility, Data Thresholds, and Decoding Strategies for Kashmiri. In Proceedings of the 14th International Conference on Recent Trends in Computing (ICRTC-2026). Springer. Best Paper Award.

@inproceedings{ayoub2026anatomy,
  title     = {Anatomy of Decoder-Only LLM Failure in Low-Resource Machine
               Translation: Tokenizer Fertility, Data Thresholds, and
               Decoding Strategies for Kashmiri},
  author    = {Ayoub, Faizan and Tigga, Neha Prerna},
  booktitle = {Proceedings of the 14th International Conference on Recent
               Trends in Computing (ICRTC-2026)},
  publisher = {Springer},
  year      = {2026},
  note      = {Best Paper Award}
}