Inconsistent handling of non-ASCII characters in encodings.normalize_encoding()

[serhiy-storchaka]

Bug report

#83518 changed handling of non-ASCII characters in encodings.normalize_encoding(), but it is still inconsistent with codecs.lookup(), and not even self-consistent. For example:

>>> import encodings
>>> encodings.normalize_encoding('a¤b')
'a_b'
>>> encodings.normalize_encoding('aæb')
'ab'
>>> encodings.normalize_encoding('a-¤')
'a'
>>> encodings.normalize_encoding('a-æ')
'a_'
>>> encodings.normalize_encoding('a-¤-b')
'a_b'
>>> encodings.normalize_encoding('a-æ-b')
'a__b'

You can even get an underscore at the end or repeated underscores in the middle.

cc @malemburg, @vstinner, @shihai1991

Linked PRs

• gh-136736: Fix handling alphanumerical non-ASCII characters in encodings.normalize_encoding() #136737

[StanFromIreland]

Please see the issue I opened yesterday #136702 and also #90666 which is a bit like a duplicate.

[StanFromIreland]

This would not be an issue if we implemented the hard ascii limitation (we currently state it in the docs) like there is for bytes (ValueError).

[serhiy-storchaka]

I agree that rejecting any non-ASCII character would also solve this issue. But this issue has more narrow scope -- it is about a defect added in #83518. This is a bug fix and can be backported.

[StanFromIreland]

This implementation will be removed when the C version is implemented.

[malemburg]

Once we have encoding normalization standardized in a single function, these issues should go away.

[serhiy-storchaka]

If we consider this a bug, it should be fixed in 3.13 and 3.14. And it is easier to do this while main does not differ too much from other branches.

On other hand, there are more questions to normalization. See #88886.