Blog

Building the Kashmiri Parallel Corpus: Methodology and Challenges

Faizan Ayoub · 4 March 2026 · 6 min read · updated 30 September 2026

  • Dataset
  • Corpus
A handwritten folio of dense Sharada-script text on aged paper
A folio of Kalhana's Rajatarangini, the twelfth-century chronicle of Kashmir's kings, in Sharada script.Kalhana (text); scribe unknown, Public domain, source

Key takeaways

  • The Kashmir AI Research corpus aggregates two existing public corpora, AI4Bharat BPCC and SMUQamar, into 124,102 Kashmiri–English pairs rather than collecting new sentences.
  • The pipeline removes exact duplicates, sides outside 5 to 1,000 characters, and rows not more than 20% Perso-Arabic, then holds out 5% for evaluation.
  • Because one source is CC-BY-NC-SA-4.0, the merged set is not redistributed; the plan is to publish the pipeline that rebuilds it from the original sources.
Contents
  1. Why aggregate instead of collecting
  2. The sources
  3. The pipeline, as the code runs it
  4. Challenges
  5. Status

A parallel corpus, text in one language aligned sentence by sentence with its translation, is the foundation of any machine translation system. For Kashmiri, the useful work was not writing new sentence pairs but bringing scattered existing ones into a single, clean, reproducible training set.

Why aggregate instead of collecting

Kashmiri has parallel data, but not in one place. Most digital Kashmiri text is social media with inconsistent spelling, scanned books and newspapers that are not machine-readable, or monolingual text with no English side. The parallel collections that do exist differ in format, domain, quality and licence. Merging two of them carefully gives a larger training set than either alone, with one fixed split that results can be compared on.

The sources

  • AI4Bharat BPCC (kas_Arab): 98,923 pairs of web-crawled news and government text; 97,920 pass our filters.
  • SMUQamar Kashmiri-English Parallel Corpus: 26,182 pairs of colloquial and educational content, all of which pass. Licensed CC-BY-NC-SA-4.0.

The sentence pairs are the work of their authors. If you use the corpus, cite them.

The pipeline, as the code runs it

  1. Deduplication: exact duplicate pairs are removed after merging.
  2. Length: each side must be 5 to 1,000 characters.
  3. Script: more than 20% of the Kashmiri side must be Perso-Arabic characters, which drops Latin-script and English-only rows.
  4. Alignment check: semantic similarity is measured on a 1,000-pair sample with multilingual sentence embeddings. It is a quality check, not a filter.
  5. Split: 117,896 training and 6,206 evaluation pairs (5%) at a fixed random seed.

The result is 124,102 pairs.

Challenges

  • Script variation: the same word can be written with different letter forms and diacritics, which exact matching treats as different sentences.
  • Code-switching: everyday Kashmiri mixes in Urdu and English, so the script filter keeps some mixed text and drops some genuine Kashmiri.
  • Quality you cannot see: filters catch broken rows, not bad translations. A sample check tells you how noisy the corpus is, not which pairs are wrong.
  • Licences travel: because one source is non-commercial and share-alike, the merged set is too. It is not ours to relicense or redistribute.
  • Training data is not a benchmark: a clean evaluation needs test sentences that no training corpus contains. For that, our follow-up work uses KashEval, a separate set of test sentences checked for overlap.

Status

The corpus trained the models in the ICRTC-2026 paper and the fine-tuned translator on this site. We do not redistribute the merged data; the plan is to publish the pipeline that rebuilds it from the original sources. Full provenance is on the dataset page.

Speak Kashmiri?

Help us find where computer translation still fails Kashmiri. Mark 25 sentences, three translations each; your answers save as you go.

Take part in the study

← All posts