Skip to content

Export the ICU line break state machine #1321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
eggrobin wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

Export the ICU line break state machine #1321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b8f1806
looking at the state table
eggrobin Feb 16, 2026
1bca334
meow
eggrobin Feb 17, 2026
536295c
Looking ahead
eggrobin Feb 17, 2026
8d3e18c
meow
eggrobin Feb 18, 2026
0d2bf81
Merge remote-tracking branch 'la-vache/main' into RoBertBastIan
eggrobin Feb 23, 2026
7b0cabe
meow
eggrobin Feb 24, 2026
d8659ea
more typical names
eggrobin Feb 24, 2026
2e17f52
meow
eggrobin Feb 24, 2026
6e17b61
meow
eggrobin Feb 24, 2026
309001c
files
eggrobin Feb 24, 2026
3869771
Test it and fix it
eggrobin Feb 25, 2026
1a7288c
Add the test
eggrobin Feb 25, 2026
162861b
meow
eggrobin Feb 25, 2026
4f2300d
Fragezeichen
eggrobin Apr 17, 2026
72d36aa
Antworte
eggrobin Apr 17, 2026
a5e0d34
Smaller tables
eggrobin Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions LineBreakClasses.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Class name ; Class definition in UnicodeSet notation
CM ; [\p{Line_Break=Combining_Mark}]
BAmEastAsian ; [\p{Line_Break=Break_After}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
LF ; [\p{Line_Break=Line_Feed}]
BK|NL ; [[\p{Line_Break=Mandatory_Break}] [\p{Line_Break=Next_Line}]]
CR ; [\p{Line_Break=Carriage_Return}]
SP ; [\p{Line_Break=Space}]
EXmEastAsian ; [\p{Line_Break=Exclamation}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
QUmPimPf ; [\p{Line_Break=Quotation}-\p{gc=Pi}-\p{gc=Pf}]
AImEastAsian|ALmEastAsianmDottedCircle|SG|XXmExtPictUnassigned ; [[\p{Line_Break=Ambiguous}-[\p{ea=F}\p{ea=W}\p{ea=H}]] [\p{Line_Break=Alphabetic}-[\p{ea=F}\p{ea=W}\p{ea=H}]-[◌]] [\p{Line_Break=Surrogate}] [\p{Line_Break=Unknown}-[\p{Extended_Pictographic=True}&\p{gc=Cn}]]]
PRmEastAsian ; [\p{Line_Break=Prefix_Numeric}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
POmEastAsian ; [\p{Line_Break=Postfix_Numeric}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
OPmEastAsian ; [\p{Line_Break=Open_Punctuation}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
CP ; [\p{Line_Break=CP}]
IS ; [\p{Line_Break=Infix_Numeric}]
HY ; [\p{Line_Break=Hyphen}]
SY ; [\p{Line_Break=Break_Symbols}]
NU ; [\p{Line_Break=Numeric}]
CLmEastAsian ; [\p{Line_Break=Close_Punctuation}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
GLmEastAsian ; [\p{Line_Break=Glue}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
QU_Pi ; [\p{Line_Break=Quotation}&\p{gc=Pi}]
BB ; [\p{Line_Break=Break_Before}]
QU_Pf ; [\p{Line_Break=Quotation}&\p{gc=Pf}]
HH ; [\p{Line_Break=Unambiguous_Hyphen}]
HL ; [\p{Line_Break=HL}]
JL ; [\p{Line_Break=JL}]
JV ; [\p{Line_Break=JV}]
JT ; [\p{Line_Break=JT}]
NSmEastAsian ; [\p{Line_Break=Nonstarter}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
AK ; [\p{Line_Break=Aksara}]
VI ; [\p{Line_Break=Virama}]
AS ; [\p{Line_Break=Aksara_Start}]
IDmEastAsianmExtPictUnassigned ; [\p{Line_Break=Ideographic}-[\p{ea=F}\p{ea=W}\p{ea=H}]-[\p{Extended_Pictographic=True}&\p{gc=Cn}]]
VF ; [\p{Line_Break=Virama_Final}]
ZW ; [\p{Line_Break=ZWSpace}]
ZWJ ; [\p{Line_Break=ZWJ}]
B2 ; [\p{Line_Break=Break_Both}]
INmEastAsian ; [\p{Line_Break=Inseparable}-[\p{ea=F}\p{ea=W}\p{ea=H}]]
WJ ; [\p{Line_Break=Word_Joiner}]
PR_EastAsian ; [\p{Line_Break=Prefix_Numeric}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
ID_EastAsian ; [\p{Line_Break=Ideographic}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
OP_EastAsian ; [\p{Line_Break=Open_Punctuation}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
CL_EastAsian ; [\p{Line_Break=Close_Punctuation}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
AI_EastAsian|AL_EastAsian ; [[\p{Line_Break=Ambiguous}&[\p{ea=F}\p{ea=W}\p{ea=H}]] [\p{Line_Break=Alphabetic}&[\p{ea=F}\p{ea=W}\p{ea=H}]]]
AL_DottedCircle ; [\p{Line_Break=Alphabetic}&[◌]]
ID_ExtPictUnassigned|EBmEastAsian ; [[\p{Line_Break=Ideographic}&[\p{Extended_Pictographic=True}&\p{gc=Cn}]] [\p{Line_Break=E_Base}-[\p{ea=F}\p{ea=W}\p{ea=H}]]]
EB_EastAsian ; [\p{Line_Break=E_Base}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
BA_EastAsian ; [\p{Line_Break=Break_After}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
NS_EastAsian|CJ ; [[\p{Line_Break=Nonstarter}&[\p{ea=F}\p{ea=W}\p{ea=H}]] [\p{Line_Break=Conditional_Japanese_Starter}]]
H2 ; [\p{Line_Break=H2}]
H3 ; [\p{Line_Break=H3}]
EX_EastAsian ; [\p{Line_Break=Exclamation}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
IN_EastAsian ; [\p{Line_Break=Inseparable}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
PO_EastAsian ; [\p{Line_Break=Postfix_Numeric}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
CB ; [\p{Line_Break=Contingent_Break}]
AP ; [\p{Line_Break=Aksara_Prebase}]
GL_EastAsian ; [\p{Line_Break=Glue}&[\p{ea=F}\p{ea=W}\p{ea=H}]]
RI ; [\p{Line_Break=Regional_Indicator}]
EM ; [\p{Line_Break=E_Modifier}]
XX_ExtPictUnassigned ; [\p{Line_Break=Unknown}&[\p{Extended_Pictographic=True}&\p{gc=Cn}]]
SAmMnmMc ; [[\p{Line_Break=Complex_Context}-\p{gc=Mn}-\p{gc=Mc}]]
SA_Mn|SA_Mc ; [[[\p{Line_Break=Complex_Context}&\p{gc=Mn}]] [[\p{Line_Break=Complex_Context}&\p{gc=Mc}]]]
201 changes: 201 additions & 0 deletions LineBreakStates.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
# State name ; Accepting (Yes, No, or lookahead name); lookahead name or empty.
START ; No ;
SP QU_Pf ; No ;
PRmEastAsian OPmEastAsian ; No ;
PRmEastAsian OPmEastAsian IS ; No ;
AK AK ; No ;
SP IS ZWJ ; No ;
CM ; Yes ;
NU ; Yes ;
HL ; Yes ;
OPmEastAsian SP CM ; Yes ;
BAmEastAsian ; Yes ;
SP ; Yes ;
EXmEastAsian ; Yes ;
CLmEastAsian ; Yes ;
QUmPimPf ; Yes ;
WJ ; Yes ;
OPmEastAsian ; Yes ;
GLmEastAsian ; Yes ;
ZWJ ; Yes ;
BAmEastAsian ZWJ ; Yes ;
EXmEastAsian ZWJ ; Yes ;
PRmEastAsian ZWJ ; Yes ;
POmEastAsian ZWJ ; Yes ;
IS ZWJ ; Yes ;
CP ZWJ ; Yes ;
HY ZWJ ; Yes ;
SY ZWJ ; Yes ;
NU ZWJ ; Yes ;
CLmEastAsian ZWJ ; Yes ;
BB ZWJ ; Yes ;
HH ZWJ ; Yes ;
HL ZWJ ; Yes ;
JV ZWJ ; Yes ;
JT ZWJ ; Yes ;
AK ZWJ ; Yes ;
IDmEastAsianmExtPictUnassigned ZWJ ; Yes ;
B2 ZWJ ; Yes ;
AL_DottedCircle ZWJ ; Yes ;
EBmEastAsian ZWJ ; Yes ;
CB ZWJ ; Yes ;
AP ZWJ ; Yes ;
XX_ExtPictUnassigned ZWJ ; Yes ;
CM HY ZWJ ; Yes ;
OPmEastAsian SP ZWJ ; Yes ;
NU CP ZWJ ; Yes ;
NU IS ZWJ ; Yes ;
NU SY ZWJ ; Yes ;
NU CLmEastAsian ZWJ ; Yes ;
HL HY ZWJ ; Yes ;
AK VI ZWJ ; Yes ;
PRmEastAsian ; Yes ;
POmEastAsian ; Yes ;
CP ; Yes ;
IS ; Yes ;
HY ; Yes ;
SY ; Yes ;
BB ; Yes ;
HH ; Yes ;
JV ; Yes ;
JT ; Yes ;
AK ; Yes ;
IDmEastAsianmExtPictUnassigned ; Yes ;
B2 ; Yes ;
OP_EastAsian ; Yes ;
GL_EastAsian ; Yes ;
AL_DottedCircle ; Yes ;
EBmEastAsian ; Yes ;
CB ; Yes ;
AP ; Yes ;
RI ; Yes ;
XX_ExtPictUnassigned ; Yes ;
CM HY ; Yes ;
CM QU_Pf ; Yes ;
OPmEastAsian SP ; Yes ;
CP SP ; Yes ;
NU CP ; Yes ;
NU IS ; Yes ;
NU SY ; Yes ;
NU CLmEastAsian ; Yes ;
HL HY ; Yes ;
JL QU_Pi ; Yes ;
JL ZWJ ; Yes ;
PR_EastAsian ZWJ ; Yes ;
ID_EastAsian ZWJ ; Yes ;
CL_EastAsian ZWJ ; Yes ;
AL_EastAsian ZWJ ; Yes ;
EB_EastAsian ZWJ ; Yes ;
BA_EastAsian ZWJ ; Yes ;
CJ ZWJ ; Yes ;
H2 ZWJ ; Yes ;
H3 ZWJ ; Yes ;
PO_EastAsian ZWJ ; Yes ;
NU CL_EastAsian ZWJ ; Yes ;
AK VI ; Yes ;
B2 SP ; Yes ;
RI ZWJ ; Yes ;
CM QU_Pf eot ; Yes ;
JL CM QU_Pi ; Yes ;
JL QU_Pf ZWJ ; Yes ;
RI RI ZWJ ; Yes ;
OPmEastAsian SP IS ZWJ ; Yes ;
RI ZWJ RI ZWJ ; Yes ;
LF ; Yes ;
CR ; Yes ;
JL ; Yes ; JL / QU_Pi JL
PR_EastAsian ; Yes ; JL / QU_Pi JL
ID_EastAsian ; Yes ; JL / QU_Pi JL
CL_EastAsian ; Yes ; JL / QU_Pi JL
CJ ; Yes ; JL / QU_Pi JL
AL_EastAsian ; Yes ; JL / QU_Pi JL
EB_EastAsian ; Yes ; JL / QU_Pi JL
BA_EastAsian ; Yes ; JL / QU_Pi JL
H2 ; Yes ; JL / QU_Pi JL
H3 ; Yes ; JL / QU_Pi JL
PO_EastAsian ; Yes ; JL / QU_Pi JL
NU CL_EastAsian ; Yes ; JL / QU_Pi JL
ZW ; Yes ; ZW / CM
ZW SP ; Yes ; ZW / CM
SP IS ; No ; SP IS / PRmEastAsian
JL CM ; Yes ; JL CM / QU_Pi JL
PR_EastAsian CM ; Yes ; JL CM / QU_Pi JL
ID_EastAsian CM ; Yes ; JL CM / QU_Pi JL
CL_EastAsian CM ; Yes ; JL CM / QU_Pi JL
CJ CM ; Yes ; JL CM / QU_Pi JL
AL_EastAsian CM ; Yes ; JL CM / QU_Pi JL
EB_EastAsian CM ; Yes ; JL CM / QU_Pi JL
BA_EastAsian CM ; Yes ; JL CM / QU_Pi JL
H2 CM ; Yes ; JL CM / QU_Pi JL
H3 CM ; Yes ; JL CM / QU_Pi JL
PO_EastAsian CM ; Yes ; JL CM / QU_Pi JL
NU CL_EastAsian CM ; Yes ; JL CM / QU_Pi JL
JL QU_Pf ; Yes ; JL QU_Pf / JL
ZW CM ; ZW / CM ;
ZW SP EXmEastAsian ; ZW / CM ;
ZW SP CLmEastAsian ; ZW / CM ;
ZW SP CP ; ZW / CM ;
ZW SP SY ; ZW / CM ;
ZW SP QU_Pf ; ZW / CM ;
ZW SP WJ ; ZW / CM ;
RI RI ; No ; RI RI / ALmEastAsianmDottedCircle
SP IS CM ; No ; SP IS CM / PRmEastAsian
SP IS PRmEastAsian ; SP IS / PRmEastAsian ;
OPmEastAsian SP IS ; Yes ; SP IS / PRmEastAsian
JL QU_Pi JL ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi PR_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi ID_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi CL_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi CJ ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi AL_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi EB_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi BA_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi H2 ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi H3 ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi PO_EastAsian ; JL / QU_Pi JL ; JL / QU_Pi JL
JL QU_Pi OP_EastAsian ; JL / QU_Pi JL ;
JL QU_Pi GL_EastAsian ; JL / QU_Pi JL ;
JL QU_Pf CM ; Yes ; JL QU_Pf CM / JL
JL QU_Pf JL ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf PR_EastAsian ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf ID_EastAsian ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf AL_EastAsian ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf EB_EastAsian ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf H2 ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf H3 ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf PO_EastAsian ; JL QU_Pf / JL ; JL / QU_Pi JL
JL QU_Pf OP_EastAsian ; JL QU_Pf / JL ;
ZW SP IS ; ZW / CM ; SP IS / PRmEastAsian
ZW SP CL_EastAsian ; ZW / CM ; JL / QU_Pi JL
ZW SP EX_EastAsian ; ZW / CM ; JL / QU_Pi JL
RI ZWJ RI ; Yes ; RI RI / ALmEastAsianmDottedCircle
RI RI CM ; No ; RI RI CM / ALmEastAsianmDottedCircle
RI RI ALmEastAsianmDottedCircle ; RI RI / ALmEastAsianmDottedCircle ;
SP IS CM PRmEastAsian ; SP IS CM / PRmEastAsian ;
OPmEastAsian SP IS CM ; Yes ; SP IS CM / PRmEastAsian
JL CM QU_Pi JL ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi PR_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi ID_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi CL_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi CJ ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi AL_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi EB_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi BA_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi H2 ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi H3 ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi PO_EastAsian ; JL CM / QU_Pi JL ; JL / QU_Pi JL
JL CM QU_Pi OP_EastAsian ; JL CM / QU_Pi JL ;
JL CM QU_Pi GL_EastAsian ; JL CM / QU_Pi JL ;
JL QU_Pf CM JL ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM PR_EastAsian ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM ID_EastAsian ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM AL_EastAsian ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM EB_EastAsian ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM H2 ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM H3 ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM PO_EastAsian ; JL QU_Pf CM / JL ; JL / QU_Pi JL
JL QU_Pf CM OP_EastAsian ; JL QU_Pf CM / JL ;
RI ZWJ RI CM ; Yes ; RI RI CM / ALmEastAsianmDottedCircle
RI ZWJ RI RI ; RI RI / ALmEastAsianmDottedCircle ; RI RI / ALmEastAsianmDottedCircle
RI RI CM ALmEastAsianmDottedCircle ; RI RI CM / ALmEastAsianmDottedCircle ;
RI ZWJ RI CM RI ; RI RI CM / ALmEastAsianmDottedCircle ; RI RI / ALmEastAsianmDottedCircle
Loading
Loading