If you work on Kashmiri language AI, you have probably found the resources scattered: datasets across Hugging Face profiles, papers across arXiv and journals, models with little documentation. This guide is a map of the main ones and what each is good for. The full, searchable list is the Resource Hub.
Why Kashmiri needs dedicated resources
Kashmiri (کٲشُر / कॉशुर, ISO 639-1 ks) is an Indo-Aryan language spoken by more than seven million people, mostly in the Kashmir Valley. It is still poorly served by mainstream AI:
- Google Translate does not support it; the systems that do (Microsoft Translator, IndicTrans2, NLLB-200) make frequent errors.
- General chat models have very little genuine Kashmiri in their training data.
- It is written in two scripts, Perso-Arabic Nastaliq (right to left) and Devanagari.
- Everyday Kashmiri mixes in Urdu, Hindi and English.
- Its morphology is rich, with pronominal clitics and complex agreement.
Datasets
Read each dataset's card before using it: licences differ, and some require the authors' permission.
| Dataset | Where | What it is |
|---|---|---|
| BPCC (kas_Arab) | AI4Bharat | About 99,000 Kashmiri–English pairs; news and government text |
| Kashmiri-English Dataset 270K | Hugging Face (SMUQamar) | The largest Kashmiri–English parallel collection |
| Kashmiri-English Parallel Corpus (26K) | Hugging Face (SMUQamar) | Colloquial and educational pairs; CC-BY-NC-SA-4.0 |
| Kashmir AI Research Parallel Corpus | Kashmir AI Research | 124,102 pairs aggregated from BPCC and SMUQamar, filtered |
| KS-LIT-3M | Hugging Face | 3.1 million words of Kashmiri text for pre-training |
| Kashmiri Wikipedia | Wikimedia | Monolingual encyclopaedia text |
| 600K-KS-OCR | Hugging Face | 602,000 synthetic Nastaliq text images for OCR |
| Kashmiri Audio Corpus (segmented) | Hugging Face | 1,955 ASR-ready speech segments |
| SLR122 | OpenSLR | Transcribed Kashmiri speech |
| FLORES-200 | Meta | Standard multilingual MT test set with Kashmiri |
| IN22 (Gen, Conv) | AI4Bharat | Indian-language MT test sets, general and conversational |
| SMOL (SmolSent) | Small, openly licensed Kashmiri–English sentence set |
For machine translation, BPCC and the SMUQamar corpora are the main training sources. For evaluation, check whether a test set overlaps with the data your model was trained on before trusting a score.
Models
| Model | Where | Use |
|---|---|---|
| IndicTrans2 (AI4Bharat) | Hugging Face | Strong open Indian-language MT, including Kashmiri |
| NLLB-200 (Meta) | Hugging Face | 200-language MT; 600M, 1.3B and 3.3B versions include Kashmiri |
| Kashmir AI Research NLLB-600M (fine-tuned) | Hugging Face | NLLB-200 fine-tuned on 93,064 pairs; behind our translator |
| IndicBERTv2 | Hugging Face | Embeddings for Indian languages |
| Kashmiri BERT | Hugging Face | A Kashmiri-specific BERT for classification and NLU |
| IndicConformer ASR | Hugging Face | Speech recognition covering Kashmiri |
| XLS-R 300M (wav2vec 2.0) | Hugging Face | Speech model to fine-tune on Kashmiri audio |
Multilingual models that do not list Kashmiri, such as mBART-50, can still be fine-tuned, but only by borrowing another language's code; start with a model that already includes Kashmiri.
Papers
| Paper | Venue | Why read it |
|---|---|---|
| IndicTrans2 (Gala et al.) | TMLR 2023 | The main open model for Indian-language MT, with Kashmiri |
| No Language Left Behind (NLLB Team) | arXiv 2022 | 200-language MT and the FLORES-200 benchmark |
| Deep neural architectures for Kashmiri–English MT | Scientific Reports, 2025 | Transformer, GRU and attention compared |
| Kashmiri news classification | Scientific Reports, 2025 | Text classification with transformer encoders |
| 600K-KS-OCR | arXiv 2026 | Large-scale Kashmiri OCR data |
| Anatomy of Decoder-Only LLM Failure (Ayoub & Tigga) | ICRTC-2026 | Our paper: a fine-tuned LLM against translation models |
Links, DOIs and many more papers are in the Resource Hub.
Tools
- Bhashini: the Government of India's language platform (translation, speech recognition, speech synthesis).
- sacreBLEU: standard MT scoring (BLEU, chrF++, TER).
- Kashmiri WordNet: a lexical database of word senses.
- Kashmir AI Research translator: Kashmiri→English in the browser.
How to get started
- Understand the language: its two scripts, spelling variation and code-switching.
- Start from a model that knows Kashmiri: IndicTrans2 or NLLB-200 zero-shot is a strong baseline.
- Fine-tune on parallel data you are allowed to use: BPCC is openly licensed; check the SMUQamar cards for their terms. Free Kaggle GPUs are enough for a 600M model.
- Evaluate honestly: use sacreBLEU and chrF++, check test sets for overlap with your training data, and get native speakers to look at the output.
- Give back: share what you build and submit it to the hub.