add a deflate compressor; gzip now interops both directions#278
Merged
Conversation
std.compress.deflate closes the other half of the project: greedy lz77 over hash chains (3-byte prefixes, 32k window, longest match up to 258 bytes, capped chain walks) encoded with the fixed huffman tables, so no tree serialization is needed. huffman codes transmit msb-first against an lsb-first bit writer, so codes are written pre-reversed — the one classic trap in the format. gzip.compress now produces real compression: a 200KB source file packs 5.2x smaller, runs compress to a third of their size, and pseudo-random input stays marginally under its input size instead of ballooning. system gunzip reads pith's output byte-identical, and a new ci check proves that on every push, alongside the existing pith-reads-system-gzip fixture. the interop regression also asserts that compression actually shrinks. limitations now states the true remaining refinement: dynamic huffman trees on the write side would shave a few more percent.
the std test suite caught what the golden checks could not: is_valid still validated through the old stored-blocks decompressor, so it called the new compressor's output invalid. it now asks decompress, and the dead stored-only path is gone. (local batteries now include test-std-self-only — ci ran it; mine did not.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
the other half of the deflate project:
std.compress.deflate— greedy lz77 over hash chains (3-byte prefixes, 32k window, 258-byte matches, capped chain walks) encoded with the fixed huffman tables (~180 lines). the classic format trap — huffman codes transmit msb-first against an lsb-first writer — is handled by pre-reversing codes.results
make gzip-interop-checkin ci, completing the loop with the existing pith-reads-system-gzip fixturehonest remainder
dynamic huffman trees on the write side would shave a few more percent — documented in limitations as the only refinement left.
what was tested
full suite green (all five golden/interop checks), fixed point byte-identical, seed refreshed, fmt + doc gates clean.