Updates to use the original ZSTD compression via C Wrapper#236
Updates to use the original ZSTD compression via C Wrapper#236mateeullahmalik merged 1 commit intomasterfrom
Conversation
Review complete. The changes look correct for aligning ZSTD compression with SDK-JS. I've flagged a minor refactoring opportunity to reduce duplication.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
There was a problem hiding this comment.
Pull request overview
This PR replaces the pure Go zstd implementation (klauspost/compress/zstd) with DataDog's C wrapper (DataDog/zstd) to ensure byte-for-byte compression compatibility with SDK-JS, which is critical for consistent RQID (content-addressed identifier) generation across client and backend services.
Key Changes:
- Switched all zstd compression calls to use
DataDog/zstdlibrary with compression level 3 - Simplified compression/decompression functions by removing concurrency configuration (handled internally by C library)
- Updated tests to use the new compression API
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/utils/utils.go | Replaced klauspost/compress/zstd with DataDog/zstd for Compress, Decompress, ZstdCompress, ZstdDecompress, and HighCompress functions; removed io and runtime imports no longer needed |
| pkg/cascadekit/ids.go | Updated generateIDFiles and generateIDs to use DataDog/zstd.CompressLevel with level 3 for RQID generation; removed encoder reuse pattern |
| pkg/cascadekit/cascadekit_test.go | Updated test compression calls to use zstd.CompressLevel(nil, data, 3) API |
| go.mod | Moved DataDog/zstd from indirect to direct dependency; moved klauspost/compress to indirect (still required by other dependencies) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![roomote-v0[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| func ZstdCompress(data []byte) ([]byte, error) { | ||
| encoder, err := zstd.NewWriter(nil) | ||
| compressed, err := zstd.CompressLevel(nil, data, 3) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to create zstd encoder: %v", err) | ||
| return nil, fmt.Errorf("failed to compress with zstd: %v", err) | ||
| } | ||
| defer encoder.Close() | ||
|
|
||
| return encoder.EncodeAll(data, nil), nil | ||
| return compressed, nil | ||
| } |
There was a problem hiding this comment.
ZstdCompress is functionally equivalent to Compress with level=3. Consider reusing Compress to reduce duplication and centralize logic.
func ZstdCompress(data []byte) ([]byte, error) {
return Compress(data, 3)
}Fix it with Roo Code or mention @roomote and request a fix.
Summary
Updates ZSTD compression implementation to use DataDog/zstd
(C wrapper) instead of klauspost/zstd (pure Go) to ensure
consistent RQID generation between SDK-JS and backend
services.
Problem
SDK-JS uses the official ZSTD C library for compression,
while the Lumera backend (chain keeper and supernode) was
using klauspost/zstd (pure Go implementation). These two
libraries produce different compressed bytes for the
same input, resulting in:
expectations
Solution
Switched to DataDog/zstd (CGO wrapper around official C
library) with compression level 3 in three critical
locations:
RQID verification
generation
functions
This ensures:
Testing
Related Issues
Fixes SDK-JS file upload/download compatibility issues.