fix: correct 6 diacritization bugs found in 83K poem corpus

[lesmartiepants]

Context

Assalamu alaikum Dr. Zerrouki,

Thank you for building and maintaining Mishkal — it is an invaluable tool for Arabic NLP. We used it to diacritize a corpus of 83,377 Arabic poems and, through systematic analysis of the output, identified several reproducible bugs. This PR contains fixes for 6 issues that fall within the Mishkal codebase itself.

Summary of Fixes

1. Duplicate diacritical marks (35,928 poems affected)

Doubled diacritics (e.g. ِِ double kasra, ْْ double sukun) appear in the output. The root cause is in alyahmor affix constant tables, but we added a post-processing safety net in _ajust_vocalized_result() using re.sub(u'([\u064B-\u0652])\\1+', u'\\1', text) to collapse duplicates.

2. Impossible hamza + sukun sequences (23,745 poems affected)

Hamza characters receive vowel+sukun, which is phonologically impossible. Added a post-processing strip in _ajust_vocalized_result().

3. Missing diacritics for unknown/foreign words (related to #2)

In unknown_tashkeel.py:vocalize_foreign(), a missing else branch caused a length mismatch between characters and marks arrays, making araby.joint() return empty strings for many common words (e.g. إبراهيم, إسماعيل, أمريكا). Added the missing branch.

4. Spurious shadda on function words (19,836 poems affected)

Common prepositions like عَلَى and إِلَى incorrectly received shadda (عَلَّى, إِلَّى), producing non-existent Arabic forms. Added targeted post-processing corrections and two unambiguous entries to CorrectedTashkeel (الى → إِلَى, أيضا → أَيْضًا).

5. Missing sun letter assimilation (2,226 poems affected)

After the definite article ال, sun letters were missing the assimilation shadda (e.g. الْنُور instead of النُّور). Added post-processing to detect الْ + sun letter and insert the shadda. Relates to existing issue #39.

6. Default word limit too low (structural)

Changed the default limit from 1,000 to 20,000 words, which covers virtually all realistic single-text inputs without requiring an explicit set_limit() call.

Files Changed

File	Changes
mishkal/tashkeel.py	Default limit 1000→20000; 5 new post-processing rules in _ajust_vocalized_result()
mishkal/tashkeel_const.py	Added 2 entries to CorrectedTashkeel dictionary
mishkal/unknown_tashkeel.py	Fixed missing else clause in vocalize_foreign()

+64 / -23 lines across 3 files.

What is NOT included

We also found bugs in the alyahmor and qalsadi dependencies:

• alyahmor: Doubled diacritics in aly_stem_noun_const.py and aly_stem_stopword_const.py prefix/suffix tables; sun letter shadda logic issue in noun_affixer.py
• qalsadi: Ya possessive/nisba confusion (ياء المتكلم vs ياء النسبة) — affects 44,594 poems but requires deeper morphological analysis changes

These will be submitted as separate PRs/issues on the respective repositories.

Approach

All fixes use conservative post-processing in _ajust_vocalized_result() to avoid changing the core analysis pipeline. This minimizes regression risk while catching the most impactful output errors. We intentionally avoided aggressive pre-corrections to words that could have legitimate alternative readings.

Notes

• Happy to split this into individual PRs per fix if that would be easier to review.
• We have a detailed investigation report documenting the root cause analysis for each issue if that would be useful.
• These fixes relate to existing issues Add the respect of rule of لا تقف العرب الا على ساكن #39, Problem in tanween  #49, and Wrong Haraka #68.

شكراً جزيلاً على عملك الممتاز في خدمة اللغة العربية.

[lesmartiepants]

Test Results

We ran a 25-test suite against both master and fix/tashkeel-bugs to verify these fixes:

Branch	Passed	Failed	Total
master	23	2	25
fix/tashkeel-bugs	25	0	25

Per-Fix Results

Fix 1 (word limit 1000 → 20000) — Clear improvement:

• master: 1260-word input truncated to 863 words (68%)
• fix: all 1260 words diacritized (100%)

Fix 2 (duplicate diacritics) — Clear improvement:

• master: الْمُجْتَهِدَتَانِِ has doubled kasra on dual feminine
• fix: single kasra الْمُجْتَهِدَتَانِ

Fixes 3-5 (hamza-sukun, function word shadda, sun letter assimilation) — Defensive safety nets:

• Both branches pass for common vocabulary (analyzer handles them correctly)
• Fixes provide post-processing guards for edge cases with unknown/foreign words
• These were observed at scale in our 83,377-poem corpus where rare words and unusual contexts triggered the bugs

End-to-End Poem Test (Al-Mutanabbi)

Both branches produce identical, correct output for classical poetry: 71 diacritical marks, no duplicates, no impossible combinations, sun letters properly assimilated.

Verdict

All fixes are safe to merge. No regressions detected. Fixes 1 & 2 have immediately measurable improvements; Fixes 3-5 are preventive guards that improve robustness at scale.

Test script: tests/test_poetry_fixes.py (25 tests covering all 5 fixes)