Skip to content

DeflateBuffer: tolerate empty chunk in feed_data without crashing on the zlib header sniff#13038

Draft
HrachShah wants to merge 1 commit into
aio-libs:masterfrom
HrachShah:fix/deflatebuffer-empty-chunk
Draft

DeflateBuffer: tolerate empty chunk in feed_data without crashing on the zlib header sniff#13038
HrachShah wants to merge 1 commit into
aio-libs:masterfrom
HrachShah:fix/deflatebuffer-empty-chunk

Conversation

@HrachShah

Copy link
Copy Markdown

What this PR does

Fixes a crash in DeflateBuffer.feed_data when the chunked transfer-encoding decoder resumes a paused stream by feeding an empty chunk. The CM-byte sniff of the RFC 1950 zlib header was reading chunk[0] on an empty bytes object and raising IndexError to the caller. The fix adds an if chunk guard so the sniff is skipped on empty input, matching the behaviour of the public StreamReader.feed_data.

Why we need it

When a chunked stream is paused (e.g. the consumer is not reading), the underlying transport can call feed_data(b"") to give the decoder another chance to drain. With an empty chunk, there are no bytes for the zlib-header sniff to inspect, but the sniff was being attempted unconditionally. The resulting IndexError: index out of range propagates out as an unhandled exception during response decoding.

Repro from the issue:

from aiohttp import StreamReader, BaseProtocol
from aiohttp.http_parser import DeflateBuffer
import asyncio

async def main():
    loop = asyncio.get_running_loop()
    protocol = BaseProtocol(loop=loop)
    reader = StreamReader(protocol, 2 ** 16, loop=loop)
    buf = DeflateBuffer(reader, "deflate")
    buf.feed_data(b"")  # IndexError: index out of range

asyncio.run(main())

How I tested it

  • New regression test test_feed_empty_data in TestDeflateBuffer exercises the exact repro.
  • Verified the test fails on the pre-fix code with IndexError: index out of range at aiohttp/http_parser.py:1142 and passes with the fix.
  • Wider run: pytest tests/test_http_parser.py -q → 405 passed, 1 pre-existing failure (test_te_header_non_ascii[py-parser], the test whose comment talks about a non-ASCII Kelvin sign but whose literal text is pure ASCII — the test only passes by happenstance of test order; predates this change).
Test log
$ AIOHTTP_NO_EXTENSIONS=1 PYTHONPATH=. pytest tests/test_http_parser.py::TestDeflateBuffer -v
... 11 passed, 1 skipped in 0.40s

Checklist

  • Added a changelog fragment under CHANGES/
  • New behaviour is covered by a test
  • Test fails on pre-fix code, passes with the fix

Drafted with Zo Bot; reviewed by HrachShah.

When a chunked transfer-encoding decoder resumes after a pause, the
underlying transport may call DeflateBuffer.feed_data(b"") to give the
decoder another chance to make progress. On a stream that has not
started decoding yet, feed_data reads chunk[0] to sniff the CM bits of
the RFC 1950 zlib header so it can decide whether to enable
suppress_deflate_header. With an empty chunk that read raises
IndexError, which surfaces to the caller as an unhandled exception
during response decoding.

Add a guard so the CM-byte sniff is skipped when chunk is empty. The
downstream decompressor.decompress_sync(b"") call already handles
empty input correctly — it was only the header-sniff branch that
crashed.
@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Jul 2, 2026
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
4849 2 4847 39
View the top 2 failed test(s) by shortest run time
tests.test_http_parser::test_te_header_non_ascii[c-parser]
Stack Traces | 0.008s run time
parser = <aiohttp._http_parser.HttpRequestParser object at 0x7f759172bac0>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_te_header_non_ascii#x1B[39;49;00m(parser: HttpRequestParser) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        #x1B[90m# K = Kelvin sign, not valid ascii.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        text = #x1B[33m"#x1B[39;49;00m#x1B[33mGET /test HTTP/1.1#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mHost: a#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mTransfer-Encoding: chunKed#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
>       #x1B[94mwith#x1B[39;49;00m pytest.raises(http_exceptions.BadHttpMessage):#x1B[90m#x1B[39;49;00m
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       Failed: DID NOT RAISE BadHttpMessage#x1B[0m

parser     = <aiohttp._http_parser.HttpRequestParser object at 0x7f759172bac0>
text       = 'GET /test HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunKed\r\n\r\n'

#x1B[1m#x1B[31mtests/test_http_parser.py#x1B[0m:862: Failed
tests.test_http_parser::test_te_header_non_ascii[py-parser]
Stack Traces | 0.008s run time
parser = <aiohttp.http_parser.HttpRequestParser object at 0x7f758f710080>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_te_header_non_ascii#x1B[39;49;00m(parser: HttpRequestParser) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        #x1B[90m# K = Kelvin sign, not valid ascii.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        text = #x1B[33m"#x1B[39;49;00m#x1B[33mGET /test HTTP/1.1#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mHost: a#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mTransfer-Encoding: chunKed#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
>       #x1B[94mwith#x1B[39;49;00m pytest.raises(http_exceptions.BadHttpMessage):#x1B[90m#x1B[39;49;00m
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       Failed: DID NOT RAISE BadHttpMessage#x1B[0m

parser     = <aiohttp.http_parser.HttpRequestParser object at 0x7f758f710080>
text       = 'GET /test HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunKed\r\n\r\n'

#x1B[1m#x1B[31mtests/test_http_parser.py#x1B[0m:862: Failed

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@codspeed-hq

codspeed-hq Bot commented Jul 2, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 83 untouched benchmarks
⏩ 83 skipped benchmarks1


Comparing HrachShah:fix/deflatebuffer-empty-chunk (838d02c) with master (f24eced)2

Open in CodSpeed

Footnotes

  1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on master (9d183e3) during the generation of this report, so f24eced was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant