Evaluator Guide — How to Evaluate Kashmiri Machine Translation

1. What are we doing?

You will be shown an original Source Sentence in Kashmiri, accompanied by a Reference Translation in English to clarify the exact intended meaning.

Below that, you will see two translations produced by two different AI models (System A and System B). Your job is to rate each system based on two main criteria: Adequacy and Fluency. Finally, you will choose an overall preference.

2. Rating Criteria

🎯 Adequacy (1 to 5)

"How much of the meaning from the original source sentence is preserved in the translation?" Ignore minor grammatical mistakes here; focus only on meaning.

5 - All Meaning: All meaning is preserved accurately.
4 - Most Meaning: Almost all meaning is preserved.
3 - Much Meaning: The core idea is preserved, but some details are lost.
2 - Little Meaning: Only small fragments of the meaning are present.
1 - None: Completely broken meaning or entirely unrelated.

🗣️ Fluency (1 to 5)

"How natural and grammatically correct does the Kashmiri translation sound?" Ignore the original meaning here; focus only on how good the Kashmiri sentence is on its own.

5 - Flawless: Perfect, native-sounding sentence.
4 - Good: Natural, with maybe one very minor phrasing issue.
3 - Non-native: Understandable, but sounds clunky or strange.
2 - Disfluent: Grammatically broken, hard to read smoothly.
1 - Incomprehensible: Complete gibberish or unreadable.

3. Overall Preference

After rating both systems individually, you must choose an overall winner:

System A or System B: Choose one if it is clearly better than the other (better meaning, better phrasing, or fewer errors).

Tie: Only choose Tie if both systems are perfectly identical, equally amazing, or equally terrible. Do not use Tie to avoid making a hard choice!

Ready? Create an Account →