blake3 hashing code optimization#219
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR consolidates BLAKE3 hashing functionality into a single location and optimizes file hashing performance. The key changes include:
- Upgrading
lukechampine.com/blake3from 1.4.0 to 1.4.1 to fix a critical hash calculation bug - Consolidating BLAKE3 hash functions from
pkg/utils/utils.goandpkg/crypto/hash.gointo a newpkg/utils/hasher.gomodule - Implementing adaptive buffer sizing for file hashing that achieves up to 7× performance improvement
Reviewed Changes
Copilot reviewed 10 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| go.mod, cmd/sncli/go.mod, tests/system/go.mod | Updated blake3 dependency to v1.4.1 |
| pkg/utils/hasher.go | New unified module with optimized BLAKE3 hashing functions |
| pkg/utils/hasher_test.go | Comprehensive test coverage for new hashing functions |
| pkg/utils/utils.go | Removed BLAKE3 functions (migrated to hasher.go) |
| pkg/crypto/hash.go | Deleted file (functionality moved to pkg/utils/hasher.go) |
| pkg/crypto/hash_test.go | Deleted file (tests moved to pkg/utils/hasher_test.go) |
| supernode/cascade/download.go, sdk/action/client.go | Updated to use new Blake3HashFile function |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
moved blake3 hash enhancements from network-maker
increased lukechampine.com/blake3 version from 1.4.0 to 1.4.1, 1.4.0 has serious bug that leads to incorrect hash calculation - "writing data in two calls where the first call ends exactly on a chunk boundary produced a wrong hash".
unified all blake3-related hash functions in pkg/utils/hasher.go
blake3 file hashing optimization:
// hashReaderBLAKE3 computes a BLAKE3 hash using an adaptive,
// manual buffered read loop to avoid the *os.File.WriteTo fast-path
// that limits throughput when using io.Copy/io.CopyBuffer.
//
// The buffer size is chosen based on data size:
//
// ≤ 4 MiB → 512 KiB buffer
// 4–32 MiB → 1 MiB buffer
// 32 MiB–2 GiB → 2 MiB buffer
// > 2 GiB → 4 MiB buffer
//
// Buffers are reused from a concurrent-safe pool to reduce allocations.
// This approach achieved the following throughput in benchmarks
// on AMD Ryzen 9 5900X (Linux, lukechampine.com/blake3):
//
// Data size | Adaptive | Manual(1MiB) | io.Copy(~32KiB)
// ----------|-------------|--------------|----------------
// 1 MiB | 1.80 GB/s | 1.26 GB/s | 0.52 GB/s
// 32 MiB | 3.00 GB/s | 3.02 GB/s | 0.50 GB/s
// 256 MiB | 3.79 GB/s | 3.35 GB/s | 0.48 GB/s
// 1 GiB | 3.91 GB/s | 3.27 GB/s | 0.53 GB/s
//
// Compared to io.Copy/io.CopyBuffer, the adaptive manual loop is
// up to ~7× faster on large files, with fewer allocations.
added unit tests for all blake3 hashing functions