Skip to content

downloads: don't use Content-Length as progress total when Content-Encoding is set#1816

Open
alliasgher wants to merge 2 commits intohttpie:masterfrom
alliasgher:fix-download-content-encoding
Open

downloads: don't use Content-Length as progress total when Content-Encoding is set#1816
alliasgher wants to merge 2 commits intohttpie:masterfrom
alliasgher:fix-download-content-encoding

Conversation

@alliasgher
Copy link
Copy Markdown

Summary

Fixes #1642.

When a server sends Content-Encoding: gzip (or any encoding), the requests library transparently decompresses the response body. The bytes written to disk are the decompressed size, which is usually much larger than the compressed Content-Length. Using Content-Length as the expected total causes downloader.interrupted to return True at the end of an otherwise successful download, producing a false "Incomplete download" error.

Before

$ http --download http://server/file.gz
Downloading 5 MB to data.json.gz
❯ Incomplete download: size=5084527; downloaded=42846965

After

Progress is shown without a denominator (same as chunked responses with no Content-Length), and no spurious error is emitted.

Fix

Skip Content-Length as the progress total whenever Content-Encoding is present.

The existing # FIXME: some servers still might send Content-Encoding: gzip comment (which referenced issue #423) is replaced by this explicit guard.

Tests

Added test_download_with_content_encoding_does_not_report_incomplete which:

  • sets Content-Length: 5 and Content-Encoding: gzip
  • writes 8 bytes of "decompressed" data (> 5 bytes)
  • asserts downloader.interrupted is False

…coding is set

When a server sends Content-Encoding (e.g. gzip), the `requests`
library transparently decompresses the response body before httpie
receives it. This means the bytes written to disk represent the
*decompressed* size, which is typically much larger than the
Content-Length header (which reflects the compressed wire size).

Using Content-Length as the expected total in that case causes the
downloader's "interrupted" check to fire at the end of an otherwise
successful download, reporting a false "Incomplete download" error.

Fix: skip Content-Length as the progress total whenever Content-Encoding
is present. Progress will be shown without a denominator, which is the
same behaviour already used for chunked responses with no Content-Length.

Fixes httpie#1642

Signed-off-by: alliasgher <alliasgher123@gmail.com>
Python 3.14 introduced an eager _check_help() call in add_argument()
that validates the help string immediately (cpython#65865).  This
causes LazyChoices.help to invoke the getter at argument-registration
time rather than only when --help is printed, breaking the old
assertion that getter.assert_not_called() after add_argument().

Expose a _ARGPARSE_CHECKS_HELP_EAGERLY flag and update the test to:
- Expect getter and help_formatter to be called once at registration on
  3.14+ (and reset their mocks so subsequent assertions stay clean).
- Keep the strict lazy-evaluation assertions for 3.13 and earlier.

Fixes httpie#1641

Signed-off-by: alliasgher <alliasgher123@gmail.com>
@alliasgher
Copy link
Copy Markdown
Author

The 2 failing tests (test_terminal_output_response_charset_detection[big5-...] and test_terminal_output_request_charset_detection[big5-...]) are in test_encoding.py and pre-date my changes — they fail on the upstream master branch too and are unrelated to downloads or cli/utils. My changes only touch httpie/downloads.py, httpie/cli/utils.py, and the corresponding test files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report: http --download misinterprets Content-Length when Content-Encoding: gzip is set

1 participant