A systematic research programme building NLP infrastructure for Roman Urdu — the dominant digital writing mode of over 230 million Urdu speakers.
Roman Urdu — Urdu written in the Latin script — is the default mode of digital communication for hundreds of millions of people across Pakistan and India. Despite this scale, it has virtually no NLP infrastructure: no large-scale annotated datasets, no standardised orthography, no validated classifiers. Every existing Urdu NLP resource is built for Nastaliq script and does not transfer.
This repository is the central hub for an ongoing research programme that is building that infrastructure from scratch. The programme currently covers sentiment analysis and fine-grained emotion recognition, with the first openly released, large-scale, human-annotated, IAA-validated datasets and classifiers for both tasks. All resources are open-source, reproducible, and archived on Harvard Dataverse for long-term accessibility.
Roman Urdu presents compounding NLP challenges absent from most other languages:
Orthographic anarchy. No standardised spelling exists. The same word appears in dozens of valid forms by fluent native speakers in the same corpus. Khushi (happiness): khushi · khushee · khuushi · khushy · khosi — all valid, all common.
Pervasive code-switching. English words, phrases, and full clauses appear mid-utterance at the word level. No existing model was trained specifically for this pattern in Roman Urdu.
No pre-existing tooling. No Roman Urdu WordNet, POS tagger, parser, or standard tokeniser exists. Everything must be built from foundation models up.
Script mismatch with standard Urdu NLP. Nastaliq-script Urdu resources — the only large Urdu corpora available — do not transfer. Roman Urdu and script Urdu are, for NLP purposes, different languages with partially overlapping vocabulary.
Stage 1 — Sentiment Foundation
──────────────────────────────────────────────────────────────────
134K Sentiment Corpus ──► Sentiment Classifier (XLM-R)
│
▼
Stage 2 — Emotion Recognition
──────────────────────────────────────────────────────────────────
28K Emotion Corpus ──► Emotion Classifier v1
(IAA-validated: │
Fleiss κ = 0.659) ▼
Emotion Classifier v2
Macro F1: 0.9896
│
▼
Stage 3 — Large-Scale Emotion Corpus [134k]
──────────────────────────────────────────────────────────────────
134K Emotion Corpus (model-labeled, large-scale release)
Each stage was built on the previous one. The sentiment corpus seeded the emotion corpus. The sentiment classifier provided the fine-tuning foundation for both emotion classifiers. The forthcoming 134K emotion corpus — labeled by the v2 model — will be the largest Roman Urdu emotion resource ever released.
| Dataset | Size | Task | Annotation | IAA | HuggingFace | Dataverse |
|---|---|---|---|---|---|---|
| RomanUrdu-NLP-Sentiment-Corpus | 134K | 3-class sentiment | Human | — | 🤗 | Harvard |
| RUEmoCorp | 28K | 7-class emotion | Human (4 annotators) | Fleiss' κ = 0.659 | 🤗 | Harvard |
| RomanUrdu-NLP-Emotion-Corpus-134K | 134K | 7-class emotion | Model-labeled | — | 🤗 | Harvard |
RUEmoCorp is the largest publicly available, IAA-validated, human-annotated emotion dataset for Roman Urdu.
Emotion classes: anger · disgust · fear · happy · neutral · sad · surprise
| Model | Task | Architecture | Macro F1 | HuggingFace |
|---|---|---|---|---|
| roman-urdu-sentiment-xlm-r | 3-class sentiment | XLM-RoBERTa base | 0.84 | 🤗 |
| roman-urdu-emotion-xlmr-v1 | 7-class emotion | XLM-RoBERTa + linear head | 0.72 | 🤗 |
| roman-urdu-emotion-xlmr-v2 | 7-class emotion | XLM-RoBERTa + 2-layer MLP | 0.9896 | 🤗 |
Full benchmark results and per-class F1 in models readmes.
from transformers import pipeline
classifier = pipeline(
"text-classification",
model="Khubaib01/roman-urdu-emotion-xlmr-v2",
trust_remote_code=True,
top_K = None
)
texts = [
"yaar dil bht dukha aaj",
"maza aa gaya bilkul",
"itna gussa aa raha hai mujhe",
"dar lag raha hai kuch bura hoga",
"yeh sun ke hairan reh gaya",
]
for text in texts:
r = classifier(text)[0]
print(f"{text!r:50} → {r['label']:10} ({r['score']:.3f})")Expected output:
'yaar dil bht dukha aaj' → sad (0.987)
'maza aa gaya bilkul' → happy (0.991)
'itna gussa aa raha hai mujhe' → anger (0.983)
'dar lag raha hai kuch bura hoga' → fear (0.976)
'yeh sun ke hairan reh gaya' → surprise (0.968)
from transformers import pipeline
classifier = pipeline(
"text-classification",
model="Khubaib01/roman-urdu-sentiment-xlm-r"
)
result = classifier("bohat acha kaam kiya hai")
print(result)
# [{'label': 'positive', 'score': 0.969}]from datasets import load_dataset
ds = load_dataset("Khubaib01/RUEmoCorp")
print(ds)
# DatasetDict({
# train: Dataset({features: ['message', 'emotion_label'], num_rows: ~24000})
# })IAA was computed on a 700-sample stratified subset (100 per class), independently re-annotated by all four annotation team members under fully blind conditions — no annotator had access to existing labels or other annotators' responses.
| Metric | Value | Landis & Koch Interpretation |
|---|---|---|
| Fleiss' Kappa (κ) | 0.6588 | Substantial agreement |
| Mean Pairwise Cohen's κ | 0.6597 | Substantial agreement |
| Full agreement (4/4) | 348 / 700 — 49.7% | — |
| Majority agreement (3/4) | 241 / 700 — 34.4% | — |
| Ambiguous (no majority) | 111 / 700 — 15.9% | — |
Fleiss' κ = 0.659 falls in the substantial agreement band (0.61–0.80, Landis & Koch 1977). For a 7-class affective labeling task in a low-resource, orthographically irregular language, this compares favourably with comparable published corpora — SemEval-2018 Task 1 (English tweet emotion) reported κ in the 0.60–0.72 range for a 4-label setup.
The dominant source of ambiguity is the anger–disgust boundary, a known challenge in affective computing for South Asian languages where both emotions share surface-level lexical forms in informal digital text.
| Title | Type | Status | Link |
|---|---|---|---|
| Roman Urdu Sentiment Classification with XLM-RoBERTa | Preprint | Published | Zenodo 18080524 |
| Roman Urdu Emotion Corpus and Classifier (full paper) | Research article | In preparation | - |
RUEC-28K was produced by a four-person expert annotation team of native Roman Urdu speakers with academic affiliations across Pakistan.
| Annotator | Affiliation | Location |
|---|---|---|
| Muzammil Shadab | Bahauddin Zakariya University (BZU) | Multan, Punjab |
| Sara | COMSATS University Islamabad (CUI) | Islamabad |
| Faiez Ahmad | Emerson University Multan (EUM) | Multan, Punjab |
| Khadija Faisal | Emerson University Multan (EUM) | Multan, Punjab |
Core Researcher, Project lead / Corresponding author: Muhammad Khubaib Ahmad, Emerson University Multan.
Emotion corpus
@data{DVN/BPWHOZ_2026,
author = {Ahmad, Muhammad Khubaib Ahmad and Khadija Faisal},
publisher = {Harvard Dataverse},
title = {{RUEmoCorp}},
UNF = {UNF:6:h03jo4SJGEAKuZCik1R/Bw==},
year = {2026},
version = {V1},
doi = {10.7910/DVN/BPWHOZ},
url = {https://doi.org/10.7910/DVN/BPWHOZ}
}Emotion Classifier
@misc{muhammad_khubaib_ahmad_2026,
author = { Muhammad Khubaib Ahmad and Khadija Faisal },
title = { roman-urdu-emotion-xlmr-v2 (Revision 7cd7dd2) },
year = 2026,
url = { https://huggingface.co/Khubaib01/roman-urdu-emotion-xlmr-v2 },
doi = { 10.57967/hf/8347 },
publisher = { Hugging Face }
}Preprint:
@misc{ahmad_2025_18080524,
author = {Ahmad, Muhammad Khubaib},
title = {Data-Centric Roman Urdu NLP: High-Quality Dataset
Curation, Privacy-Preserving Embeddings, and
State-of-the-Art Model Benchmarking
},
month = dec,
year = 2025,
publisher = {Zenodo},
version = {0.1.0},
doi = {10.5281/zenodo.18080524},
url = {https://doi.org/10.5281/zenodo.18080524},
}All datasets and models are released under the Apache 2.0 License. Free for academic and commercial use with attribution.
Muhammad Khubaib Ahmad
AI Research Engineer · Emerson University Multan · Multan, Punjab, Pakistan
- 🤗 HuggingFace: Khubaib01
- 📄 Preprint: Zenodo 18080524
- 💬 Questions, issues, or collaboration proposals: open a GitHub Issue
Building the NLP infrastructure that Roman Urdu deserves.