Skip to content

match chunked as the final coding in is_chunked_transfer_encoding#2487

Closed
metsw24-max wants to merge 2 commits into
yhirose:masterfrom
metsw24-max:chunked-final-coding-detection
Closed

match chunked as the final coding in is_chunked_transfer_encoding#2487
metsw24-max wants to merge 2 commits into
yhirose:masterfrom
metsw24-max:chunked-final-coding-detection

Conversation

@metsw24-max

Copy link
Copy Markdown
Contributor

the server and streaming client only treat a message as chunk-framed when the whole Transfer-Encoding value equals "chunked", so a message whose final coding is chunked but which names another coding first ("gzip, chunked", valid under RFC 9112 6.1 where chunked must be the final coding) is missed. is_chunked_transfer_encoding matches the entire field against "chunked", and open_stream repeats the check case-sensitively with a raw ==. on the server such a request is read as bodiless and its body is left in the socket for a keep-alive connection to parse as the next request; on the client a "Transfer-Encoding: Chunked" response desyncs the stream the same way. this reworks is_chunked_transfer_encoding to compare the last coding token of the last Transfer-Encoding line case-insensitively (RFC 9110 5.3 folds repeated lines into one ordered list) and routes open_stream through it so both paths agree. a unit test covers the helper.

@yhirose

yhirose commented Jul 11, 2026

Copy link
Copy Markdown
Owner

@metsw24-max thank you, but this PR causes CI errors.

Headers is an unordered_multimap, whose iteration order for repeated
keys is unspecified, so walking equal_range to pick the last line was
unreliable and failed CI on libstdc++. Read the Transfer-Encoding value
and match its last comma-separated coding token instead. Drop the
multi-line ordering case from the unit test, since the container cannot
represent received order across duplicate field lines.
@metsw24-max

Copy link
Copy Markdown
Contributor Author

Sorry about that. The failing assertion was one I added: it fed two separate Transfer-Encoding lines and expected the last-received one to win, but Headers is an unordered_multimap, so the order of duplicate keys isn't defined and libstdc++ iterates them the other way round from libc++. I've dropped that case and the helper now just matches the last coding token of the Transfer-Encoding field value, which is the part that actually mattered for the smuggling case. Passes locally on the no-tls and split builds now.

@yhirose

yhirose commented Jul 18, 2026

Copy link
Copy Markdown
Owner

@metsw24-max Thanks again for spotting this. I picked up the fix and opened #2500, which keeps your core idea (match chunked as the final coding, and route open_stream through the same helper) but reworks the implementation enough that a side-by-side diff is clearer than a review comment here.

The points that led me to a different implementation:

  • Only the first line is read. get_header_value(headers, "Transfer-Encoding", "", 0) returns index 0, so a message split across multiple Transfer-Encoding lines (Transfer-Encoding: gzip / Transfer-Encoding: chunked, RFC 9110 §5.3) isn't handled.
  • Duplicate-key order isn't portable. This is why the multi-line test had to be dropped: Headers is an unordered_multimap, and the iteration order of equal keys differs between libstdc++ and libc++. So "the last received line wins" cannot be determined reliably from the container.
  • Description vs. code. The PR body says it matches "the last coding token of the last Transfer-Encoding line", but the code only inspects the first line — the multi-line part isn't actually covered.

#2500 handles the multi-line case deterministically instead: it scans every Transfer-Encoding line and, when more than one is present (where the true final coding is unknowable), fails safe by treating any field that names chunked as chunked — a mis-parse only closes the connection, whereas the opposite error is the smuggling case. It's credited to you in the description. Let's continue the discussion over there.

@yhirose yhirose closed this Jul 18, 2026
yhirose added a commit that referenced this pull request Jul 18, 2026
is_chunked_transfer_encoding compared the whole Transfer-Encoding field
value against "chunked", so a message whose final coding is chunked but
which names another coding first ("gzip, chunked", valid under RFC 9112
6.1) was read as unframed. On a keep-alive server the body was then left
in the socket and parsed as a smuggled request; open_stream repeated the
check case-sensitively with a raw ==, desyncing the client stream the
same way.

Rework the helper to match the last coding token case-insensitively and
route open_stream through it so both paths agree. The codings may also be
split across multiple Transfer-Encoding lines (RFC 9110 5.3); since
Headers is an unordered_multimap whose duplicate-key iteration order is
not portable, the final coding of a multi-line field cannot be
determined reliably, so treat any such message that names chunked as
chunked (fail safe: a mis-parse only closes the connection, whereas the
opposite error enables smuggling). A unit test covers the helper,
including order-independent multi-line cases.

Based on #2487 by @metsw24-max.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants