Skip to content

Fix compression change#33

Merged
jm-rivera merged 2 commits intomainfrom
fix-compression-change
Apr 15, 2026
Merged

Fix compression change#33
jm-rivera merged 2 commits intomainfrom
fix-compression-change

Conversation

@jm-rivera
Copy link
Copy Markdown
Collaborator

This pull request adds support for reading Deflate64-compressed ZIP files, which are now used by the OECD for bulk data downloads. Since Python's standard zipfile module does not natively support Deflate64, the package now patches zipfile at runtime using the inflate64 library to handle these files transparently. The update also includes new tests to ensure Deflate64 support works correctly, and adds inflate64 as a dependency.

Deflate64 ZIP Support:

  • Added a new module (src/oda_reader/download/_deflate64.py) that monkey-patches Python's zipfile to transparently support Deflate64 (compression type 9) for decompression using the inflate64 library. Only reading is patched; writing Deflate64 ZIPs is not supported.
  • The patch is imported in src/oda_reader/download/download_tools.py to ensure Deflate64 support is enabled for all downloads.

Dependency Management:

  • Added inflate64 (>=1.0.0) to the list of dependencies in pyproject.toml.
  • Updated the package version to 1.5.1 to reflect the new feature.

Testing:

  • Added comprehensive unit tests in tests/download/unit/test_deflate64.py to verify Deflate64 ZIP reading, including roundtrip tests and integration with the existing download tools for CSV, Parquet, and TXT files.

Documentation:

  • Updated CHANGELOG.md to document Deflate64 support and the new dependency.

@jm-rivera jm-rivera merged commit d09ea09 into main Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant