`codecs.encode` with `utf-*` encoding and errors returing `str` rejects surrogates blindly

[litlighilit]

Bug report

Bug description:

For codecs.encode,
with utf-* encoding, and a custom errors which returns str,
if you pass some characters that are not valid UTF characters (e.g. surrogates),
UnicodeEncodeError is just raised and there's not the expected (and documented) case
where the returned str is appended.

import codecs

ERRORS_NAME = "returning non-ascii"

# something being not encod-able via `utf-*`
BAD_UTF = "\uD800"  # the first high surrogate character


def register_repl_error(repl: str):
    def error_handle(exc: UnicodeEncodeError) -> tuple[str, int]:
        return (repl, exc.end)

    codecs.register_error(ERRORS_NAME, error_handle)


def encode_surrogate(encoding: str, repl: str):
    register_repl_error(repl)
    max_enc_len = 9
    pre = f"codecs.encode({BAD_UTF!r}, {encoding=:{max_enc_len}}) "
    try:
        res = codecs.encode(BAD_UTF, encoding, ERRORS_NAME)
    except UnicodeEncodeError as err:
        reason = err.reason
        print(pre + f"raises with {reason=}")

    else:
        print(pre + f"returns {res}")


NON_ASCII = "龍"  # loong in Chinese

## utf-*

for i in ('8', '16', '32', '16-le', '16-be'):
    encode_surrogate("utf-" + i, NON_ASCII)

print('-'*3)


# The following is some non-utf* encoding, which works fine

## cjk

### zh
for enc in ("gbk", "big5"):
    encode_surrogate(enc, NON_ASCII)

### jp
for enc in ("Shift_JIS", "EUC-JP"):
    encode_surrogate(enc, NON_ASCII)

Output:

codecs.encode('\ud800', encoding=utf-8    ) raises with reason='surrogates not allowed'
codecs.encode('\ud800', encoding=utf-16   ) raises with reason='surrogates not allowed'
codecs.encode('\ud800', encoding=utf-32   ) raises with reason='surrogates not allowed'
codecs.encode('\ud800', encoding=utf-16-le) raises with reason='surrogates not allowed'
codecs.encode('\ud800', encoding=utf-16-be) raises with reason='surrogates not allowed'
---
codecs.encode('\ud800', encoding=gbk      ) returns b'\xfd\x88'
codecs.encode('\ud800', encoding=big5     ) returns b'\xc0s'
codecs.encode('\ud800', encoding=Shift_JIS) returns b'\x97\xb4'
codecs.encode('\ud800', encoding=EUC-JP   ) returns b'\xce\xb6'

CPython versions tested on:

3.9, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux, Windows

Linked PRs

• gh-127305: more detailed doc for register_error's error_handler #139554

[litlighilit]

I'm going to fix it until I realize something serious to communicate: how errors shall be handled within error_handle?

One is simply re-call encode with 'strict' errors against the returned str, which,

however, may lead to dead loop (recursion), because `encode` will still invoke error_handler against the same data of `str`

is a method also used by _multibytecodec module.

(EDIT:using strict as fallback won's cause deadloop)

[picnixz]

I'll have a look at it in a few days. I think error handler should not have errors but I need to verify. If the error handler errs, then the exception should be propagated.

[litlighilit]

I'll have a look at it in a few days. I think error handler should not have errors but I need to verify. If the error handler errs, then the exception should be propagated.

Right, and it in fact does as expected, as edited above, my previous option was incorrect.

---

Then let's return to the original issue:

I've figured out where the mistaken code lies, and there're at least three places required to be fixed.

Yet these two days I'm too busy to focus on this patch, sorry in advance but I'll make it this weekday.

[StanFromIreland]

The current behaviour is actually tested:

cpython/Lib/test/test_codeccallbacks.py

Lines 815 to 827 in ff0cf0a

	for enc, input, repl in (
	("utf-8", "[\udc80]", "\U0001f40d"),
	("utf-16", "[\udc80]", "\U0001f40d"),
	("utf-32", "[\udc80]", "\U0001f40d"),
	):
	with self.subTest(encoding=enc):
	with self.assertRaises(UnicodeEncodeError) as cm:
	input.encode(enc, "test.replacing")
	exc = cm.exception
	self.assertEqual(exc.start, 1)
	self.assertEqual(exc.end, 2)
	self.assertEqual(exc.object, input)

cc @serhiy-storchaka

[litlighilit]

The current behaviour is actually tested:

cpython/Lib/test/test_codeccallbacks.py

Lines 815 to 827 in ff0cf0a

	for enc, input, repl in (
	("utf-8", "[\udc80]", "\U0001f40d"),
	("utf-16", "[\udc80]", "\U0001f40d"),
	("utf-32", "[\udc80]", "\U0001f40d"),
	):
	with self.subTest(encoding=enc):
	with self.assertRaises(UnicodeEncodeError) as cm:
	input.encode(enc, "test.replacing")
	exc = cm.exception
	self.assertEqual(exc.start, 1)
	self.assertEqual(exc.end, 2)
	self.assertEqual(exc.object, input)

cc @serhiy-storchaka

The test is not the case in which the issue talks.

As in these tests, what the errors handler returns is a valid char ( \U0001f40d, i.e. '🐍') encodable via given encoding

[litlighilit]

I just realize the only issue is we haven't documented what errors will we use for the returned string.

What we have been doing is using 'strict'

May we just add a warning/note section to doc of register_error

[serhiy-storchaka]

The original test was added in 9ab7dd4. I only significantly extended it in 18b07d7.

Why is the non-ASCII replacement an error? I guess just for code simplicity and because there wasn't a use case for this. In Python 2, 8-bit strings and Unicode strings are interchangeable if they contain only ASCII characters. That's why it made sense to support such Unicode strings.

If Asian multibyte codecs use recursive encoding for replacement string, then this can be done for other codecs as well. But there are pitfalls: UTF-16 and UTF-32 codecs without the byteorder qualifier produce a BOM -- but we don't want to insert a BOM in the middle of string.