Server-side histogram binning + grid/sort perf fixes#221
Open
AlexGrayBox wants to merge 2 commits into
Open
Conversation
AlexGrayBox
force-pushed
the
histogram-server-binning
branch
from
8f4e68a to
59a5776
Compare
- GetHistogram RPC: bin one column server-side into <=512 typed bins (min/max/avg/count + per-(origin,discarded) sub-bars) instead of the client pulling every row and binning in the browser. Bit-identical to the client binning; ~116x smaller payload, ~50ms warm. Adds the proto messages + RPC, regenerated pb2/pb2_grpc, DataService.GetHistogram, and servicer delegation. - ApplyDataQuery: skip the forced full-view rebuild for SORT-ONLY operations (a sort just re-orders the existing snapshot). Global sort ~7.5s -> ~0.5s. - _slowUpdateInternals: run the view rebuild on a background thread for reader-triggered (non-force) refreshes, so grid/histogram reads never block on the multi-second collapse+combine. Reader p95 ~3000ms -> ~130ms. Filters/ resets still refresh inline (need fresh data). - ws-classification example: loosen eval (100->500) / checkpoint (25->250) cadence and use a bigger eval batch (16->128) so eval stops dominating wall-clock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AlexGrayBox
force-pushed
the
histogram-server-binning
branch
from
59a5776 to
59318e9
Compare
guillaume-byte
changed the base branch from
dev
to
v1.2.3---UI-Optimization-and-new-features
…am-server-binning
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Moves histogram binning to the server behind a typed
GetHistogramRPC, and fixes grid reader/sort stalls during active training.Histogram (server-side binning)
GetHistogramRPC + typedHistogramBin/HistogramSubBar/HistogramRequest/HistogramResponseproto messages (regenerated_pb2/_pb2_grpc).DataService.GetHistogram: equal-population positional binning over the view df; returns<= max_binstyped bins with per-bin min/max/avg/count + origin/discarded sub-bars.Grid / sort perf
ApplyDataQuerysort fast-path: sort-only queries skip the forced full_slowUpdateInternalsrebuild (~7.5–15.5s → ~0.5s under training)._bg_view_refreshon aWL-ViewRefreshthread +_refresh_in_flightlock): readers no longer block on the 2–4s rebuild (p95 ~3000ms → ~130ms).Example config
Notes
.py;data_service.py/trainer_services.pyare excluded by the CI path filter,test_grpc_user_actions.py(the one lintable change) is clean.🤖 Generated with Claude Code