Skip to content

Simplify decode() and validate UTF-8 continuation bytes#17

Merged
yhirose merged 2 commits into
masterfrom
improve-decode-validate-utf8
Jun 10, 2026
Merged

Simplify decode() and validate UTF-8 continuation bytes#17
yhirose merged 2 commits into
masterfrom
improve-decode-validate-utf8

Conversation

@yhirose

@yhirose yhirose commented Jun 10, 2026

Copy link
Copy Markdown
Owner

Summary

Builds on #15 by @samhocevar, with one correctness fix added.

  • Simplify & speed up decode() — remove the internal for_each() helpers (UTF-8 and UTF-16) and call decode_codepoint() directly. This drops the duplicated byte scan and speeds up decoding by roughly 20% (UTF-16) and 30% (UTF-8). The helpers were used nowhere outside decode().
  • Validate UTF-8 continuation bytes in decode_codepoint(). Without the for_each() grouping, an invalid lead byte would consume the following bytes unconditionally — they were only masked with & 0x3F, never checked — and swallow trailing valid characters. Rejecting bytes that are not 0x800xBF makes decode() resync byte-by-byte and recover the following text.

Why the validation is needed

The bare simplification in #15 changes behaviour on invalid UTF-8 (valid input is unaffected):

input before this fix with continuation validation ICU / Python / Rust / browsers
E0 41 42 B (swallows A, bogus cp) AB <U+FFFD>AB
F0 'Hello' <U+896C>lo (swallows Hel) Hello <U+FFFD>Hello
E1 80 41 <U+1001> (swallows A) A <U+FFFD>A

Every mainstream decoder validates continuation bytes and never swallows the following valid characters. This change brings the resync behaviour in line with them. (The library still silently drops invalid bytes rather than emitting U+FFFD — that existing design choice is preserved; this PR is not a full Table 3-7 validator.)

Tests

Added regression tests (test/test.cpp):

  • decode invalid utf8 resync — invalid lead bytes resync and recover the following text instead of swallowing it.
  • decode_codepoint rejects bad continuation bytesdecode_codepoint() returns 0 on bad continuation bytes, and still accepts well-formed sequences.
  • utf16 decode invalid resync — a lone high surrogate resyncs without swallowing the next unit.

These fail against the un-validated decode() and pass with the fix.

Verification

  • Full test suite (including the new cases): all pass.
  • ~500k random valid scalars, encode → decode round-trip: 0 failures (UTF-8 and UTF-16), so no regression on valid input.
  • 100k fuzzed byte sequences vs Python's standard decoder (errors='replace', U+FFFD stripped): 98.3% exact match; every remaining difference is the library's pre-existing permissiveness (overlong / surrogate-encoded / out-of-range), with no unexplained mismatches.
  • Builds clean under -Wall -Wextra -Wshadow.

yhirose added 2 commits June 9, 2026 23:16
Remove the internal for_each() helpers and let decode() call
decode_codepoint() directly. This drops the duplicated scan that
for_each() and decode_codepoint() each performed, simplifying the code
and speeding up decoding (~20% for UTF-16, ~30% for UTF-8). The helpers
were used nowhere outside decode(). Based on #15 by Sam Hocevar.

Also validate continuation bytes in the UTF-8 decode_codepoint(). Without
the for_each() grouping, an invalid lead byte would otherwise consume the
following bytes unconditionally (they were only masked, never checked),
swallowing trailing valid characters -- e.g. "\xE0AB" decoded to a single
bogus code point instead of "AB". Rejecting bytes that are not 0x80-0xBF
makes decode() resync byte-by-byte and recover the following text, matching
the behaviour of ICU, Python, Rust and browsers (minus U+FFFD emission).
The UTF-16 path already validated surrogate pairs and is unchanged in
behaviour.
Cover that an invalid UTF-8 lead byte resyncs at the next byte instead of
swallowing the following valid characters, that decode_codepoint() reports
failure on bad continuation bytes, and the matching lone-surrogate resync
for UTF-16. These fail against the un-validated decode and pass with it.
@yhirose
yhirose merged commit 7264a28 into master Jun 10, 2026
6 checks passed
@yhirose
yhirose deleted the improve-decode-validate-utf8 branch June 10, 2026 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant