cli: support --list for gzip files - #4713
Open
RachitJava wants to merge 1 commit into
Open
Conversation
zstd -l on a gzip file previously failed with "File not compressed by zstd", which is a gap in gzip drop-in compatibility when zstd is used through a gzip symlink (facebook#3037). Teach the --list command to recognize gzip files: parse the gzip header (including the optional FEXTRA, FNAME, FCOMMENT and FHCRC fields), then read the CRC32 checksum and the decompressed size from the trailer. No decompression is performed, and no zlib support is required, so this works in all build configurations. As with gzip -l, the reported decompressed size is modulo 2^32 and accounts only for the last stream of a multi-stream file; this limitation is documented in the man page. The Check column reports CRC32 for gzip files, and the verbose view displays a gzip-specific summary. Files that are neither zstd nor gzip are still rejected as before. Fixes facebook#3037
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3037
zstd -lon a gzip file previously failed withFile "..." not compressed by zstd, which is a gap in gzip drop-in compatibility when zstd is used through agzipsymlink. Per the discussion in #3037,-lsupport for gzip files is the only remaining request in that issue.What this does
The
--listcommand now recognizes gzip files: it parses the gzip header (including the optionalFEXTRA,FNAME,FCOMMENTandFHCRCfields), then reads the CRC32 and the decompressed size (ISIZE) from the trailer. No decompression is performed and no zlib is required, so this works in all build configurations — verified withHAVE_ZLIB=0 HAVE_LZMA=0 HAVE_LZ4=0.The verbose view (
-lv) prints a gzip-specific summary (format, sizes, ratio, CRC32) instead of the zstd-specific fields (frames, dict ID, window size), and the totals row reportsCRC32when all listed files are gzip.Semantics
Same as
gzip -l: the decompressed size comes from the trailer, so it is modulo 2^32 and accounts only for the last stream of a multi-stream file. This limitation is documented in the man page entry for-l. Malformed inputs (truncated files, reserved header flags, unknown compression methods) are rejected with an error, and files that are neither zstd nor gzip fail with the same message as before.Tests
Added
tests/cli-tests/list/list-gzip.shwith a deterministic embedded gzip fixture, covering the default and verbose listings, multi-file totals, the not-compressed error path, and truncated input. The full cli-tests suite passes (45/45). Also compared the output againstgzip -lon the same files, including a concatenated multi-stream file and a file usingFHCRC.