Parallel corpus

Kashmiri–English, 124,102 pairs

Two existing public corpora, merged, deduplicated and filtered so that models can be trained and compared on the same data. The sentence pairs are the work of AI4Bharat and S. M. U. Qamar; the aggregation, filtering and split are ours.

A page of a Sanskrit manuscript written in brown ink in Sharada script
A Sanskrit manuscript in Sharada, the script once used across Kashmir, acquired in the 19th century.Ms Sarah Welch (photograph), CC BY-SA 4.0, source
124,102
Sentence pairsafter filtering
97,920
From AI4Bharat BPCCof 98,923 before filtering
26,182
From the SMUQamar corpusall kept by the filters
117,896 / 6,206
Train / evaluationfixed seed

Where the data comes from

None of these sentence pairs originate here. They were collected and aligned by others, and this corpus exists on top of their work. If you use it, cite them.

AI4Bharat BPCC (kas_Arab)
98,923 pairs of web-crawled news and government text; 97,920 pass the filters.AI4Bharat, IIT Madras · huggingface.co/datasets/ai4bharat/BPCC
SMUQamar Kashmiri-English Parallel Corpus
26,182 pairs of colloquial and educational content, all of which pass the filters. Licensed CC-BY-NC-SA-4.0.S. M. U. Qamar · huggingface.co/datasets/SMUQamar/Kashmiri-English-Parallel-Corpus

What we did to it

Five steps, in this order, each done by a script in the research repository.

  1. Deduplication

    Exact duplicate pairs removed after merging the two sources.

    kashmiri_dataset_aggregator.py

  2. Length

    5 to 1,000 characters on each side.

    quality_check.py

  3. Script

    More than 20% Perso-Arabic characters on the Kashmiri side, so Latin-script and English-only rows drop out.

    quality_check.py

  4. Alignment check

    Semantic similarity is measured on a 1,000-pair sample with multilingual sentence embeddings, as a quality check. It is not used to remove pairs.

    quality_check.py

  5. Split

    117,896 training and 6,206 evaluation pairs (5%) at a fixed random seed.

    prepare_data.py

Licence and availability

The SMUQamar corpus is non-commercial and share-alike (CC-BY-NC-SA-4.0), so any set that includes it inherits those terms. It is not ours to relicense, and we do not redistribute the merged data. The plan is to publish the pipeline that rebuilds it (aggregation and filtering code, source manifests and the split seed), so anyone can reproduce the identical 124,102 pairs from the original sources.

Why it exists, and what it is not

What Kashmiri parallel data exists is scattered across collections with different formats, quality and licences. This corpus adds no new sentence pairs: it standardises two of them into one training set with a fixed split. It is what the ICRTC-2026 models were trained and tested on.

It is a training corpus, not a clean benchmark. For evaluation, our follow-up work uses KashEval, a separate set of test sentences checked for overlap with public training corpora.

Help us check the translations

Native Kashmiri speakers can mark computer translations in our current study: three translations per sentence, scored and checked for mistakes.

Take part