Evaluator Guide
How to rate the machine translation outputs correctly.
1. What are we doing?
You will be shown an original Source Sentence in Kashmiri, accompanied by a Reference Translation in English to clarify the exact intended meaning.
Below that, you will see two translations produced by two different AI models (System A and System B). Your job is to rate each system based on two main criteria: Adequacy and Fluency. Finally, you will choose an overall preference.
2. Rating Criteria
Adequacy (1 to 5)
"How much of the meaning from the original source sentence is preserved in the translation?" Ignore minor grammatical mistakes here; focus only on meaning.
- 5 - All Meaning: All meaning is preserved accurately.
- 4 - Most Meaning: Almost all meaning is preserved.
- 3 - Much Meaning: The core idea is preserved, but some details are lost.
- 2 - Little Meaning: Only small fragments of the meaning are present.
- 1 - None: Completely broken meaning or entirely unrelated.
Fluency (1 to 5)
"How natural and grammatically correct does the Kashmiri translation sound?" Ignore the original meaning here; focus only on how good the Kashmiri sentence is on its own.
- 5 - Flawless: Perfect, native-sounding sentence.
- 4 - Good: Natural, with maybe one very minor phrasing issue.
- 3 - Non-native: Understandable, but sounds clunky or strange.
- 2 - Disfluent: Grammatically broken, hard to read smoothly.
- 1 - Incomprehensible: Complete gibberish or unreadable.
3. Overall Preference
After rating both systems individually, you must choose an overall winner:
System A or System B: Choose one if it is clearly better than the other (better meaning, better phrasing, or fewer errors).
Tie: Only choose Tie if both systems are perfectly identical, equally amazing, or equally terrible. Do not use Tie to avoid making a hard choice!