Conversation
- Fix hatch_build.py: use force_include (underscore) not force-include (hyphen) so hatchling actually includes the compiled binary in the wheel - Set pure_python=False and platform tag (manylinux_2_17) so pip installs the correct architecture-specific wheel - Rewrite release.yml with matrix builds (x86_64 + aarch64) producing one wheel per arch, separate sdist, combined publish step - Attach arch-suffixed pwalk2 binaries to GitHub releases Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch from ThreadPoolExecutor to raw mp.Process workers to eliminate GIL contention from botocore XML parsing that was starving the main thread. Workers now use a work-stealing pattern with blocking queues and None sentinels for clean shutdown. Also adds --max-objects CLI option for benchmarking, progress display during prefix discovery, and per-worker timing instrumentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate parquet files and meta.json under a dashboard_data/ subdirectory alongside dashboard.html, using _export_dashboard_data for both build and update commands. Update test helpers accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old approach used the shallowest file's parent directory as root, which breaks when files exist at different depths (e.g. S3 data). Use lexicographic min/max to compute the LCP in O(1) comparisons. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Folder chip sizes were computed from cell.folder (immediate parent dir only), giving values much lower than compute_top_folders which explodes all path components. Now accumulates chip sizes for every matching folder tag in the node's path, consistent with Python-side computation. Folder filtering now checks both node path parts and cell.folder, catching files in pruned subdirectories that the node-only approach missed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Compares two .feather scan files and builds a diff dashboard showing where disk usage grew or shrank. Merges pruned trees via outer join, builds merged cubes with old/new/delta columns, and produces a standalone HTML dashboard with diverging red/green treemap coloring based on absolute delta, deep linking support, and top changes panel. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ith an incremental view
The train partition was renamed/retired; point the scan job at gpu. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalize the scan command beyond a single path/bucket: - --modal scans Modal volumes by fanning out one pwalk2 Modal job per volume (PATH = one volume, or omit for all). New modaljob.py. - --s3 without a PATH scans every visible bucket into one capture, each prefixed by /<bucket>. New scan_all_buckets() in s3scan.py. - --s3/--modal are mutually exclusive; PATH is required only for local and single-bucket/volume scans. - -w/--workers now defaults per-backend (S3 32 / Modal 8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add foldernorm.normalize_columns: each rule is a regex whose capture groups are collapsed to '*' (the rest preserved), applied per path component over the unique values and mapped back vectorially. This groups date-stamped/numbered folders (delivery-20260423 -> delivery-*) before tree/cube/folder aggregation so they don't blow up cardinality. Wired through build(collapse_rules=...) and exposed as the repeatable --collapse REGEX option on both `scan` and `dashboard`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When scanning all volumes/buckets the common-prefix root is "" and its children's parent is also "" (falsy), so buildIndices' truthiness check orphaned every top-level node and collapsed the totals. Detect the root by index (TREE[0]) instead. Also cache-bust the parquet/meta fetches per page load so a rebuilt dashboard is always picked up (skipped on file:// which has no query strings). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The treemap sized boxes by RECURSIVE_ABS_DELTAS (sum of |per-cell delta| down the subtree), so a directory that reshuffles data between siblings (e.g. one subdir +1.6 TB, another -1.6 TB) showed enormous churn while its net change was tiny, drowning out the real net contributors. Size boxes by |net delta| instead. Net sizing can violate Plotly's branchvalues:'total' invariant when siblings cancel, so a top-down traversal only expands a node's children when their magnitudes fit within its net; otherwise the node is collapsed to a leaf and flagged as a reshuffle (⇄), with its churn shown on hover. The view root is always expanded so drilling into a reshuffle subtree still reveals it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Describe ducl's three-language split (C pwalk2 listing, Python orchestration/build, browser dashboard) and the scan->build->dashboard data flow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reorders a scan feather into DFS order so each directory's subtree is a contiguous [dir_offset, enddir_offset) row range, emitting one summary row per directory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thins out <prefix>-YYYY-MM-DD captures (and their sidecars/dashboards): keeps everything recent, then one capture per exponentially growing age bucket. Never drops the newest or anything a latest_* symlink points at. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single-process prototype of the epoch/barrier architecture for building rm/cp/rsync on top of a parallel walker, with an out-of-order executor so correctness rests only on epoch ordering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rashishhume
approved these changes
Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.