feat: WIP multi-step scheduling continuation#46
Open
define-null wants to merge 3 commits into
Open
Conversation
define-null
force-pushed
the
defnull/net-681-multi-step-scheduling-3
branch
5 times, most recently
from
f7758cb to
74dd9a3
Compare
define-null
changed the base branch from
defnull/net-681-multi-step-scheduling-2
to
master
define-null
force-pushed
the
defnull/net-681-multi-step-scheduling-3
branch
from
814ecd6 to
5f0a965
Compare
…uction-shaped data. Three tracks: fix two correctness bugs, push the per-cycle heavy lifting into Postgres, and consolidate the hash-ring code both schedulers duplicated. The reshuffle simulator gains the dataset operations and metrics to measure it. **Correctness** - **Weight-strategy desync**: the pk and current-placement vectors were built from the original chunk order while the scheduler input came from the weight strategy's filtered/reordered list — placements got attributed to the wrong chunk and uncovered chunks were still scheduled. Both now derive from the same filtered list. - **Phantom assignment rows**: the worker tombstone's FK forced an assignment id to be minted in Phase A, leaking an unused row on every Phase B shortage rollback. Replaced with a bare drop-tick (no FK); id minting moves to Phase B. **Postgres performance & scale** - **Twin-table set-diffing**: the app-side reconcile becomes a twin ideal table diffed in pure SQL and promoted by a transactional rename; several fetches and post-write phases collapse into single round-trips. - **GiST range index for the overlap probe**: replaces the O(N²) nearest-left btree walk on new datasets. - **chunk replace/copy fast paths**: batch corrections in one transaction, O(P) readiness resolution, server-side copy, cycle-scoped cleanup. - **Server-side routing visible set**: drops the large per-cycle visible-pk array upload; the set is derived from the same metadata predicate server-side. - **Leaner IdealMapping**: per-chunk `HashSet` + outer `BTreeMap` become a positional `Vec`, canonicalized only at the COPY boundary. **Shared ring machinery**: extracted into `src/rings.rs`; `WorkerRingCache` rebuilds rings only when the worker id list changes, so a steady fleet reuses them across cycles. **Per-dataset schema**: a `schemas` table plus `chunks.schema_id` lets files be derived from the schema instead of stored per-chunk; adds a `tables_present` bitmap and per-pipeline chunk types so the hot read decodes a slim row. **Reshuffle simulator**: step-triggered copy/replace sourced from live storage; ideal-vs-ideal shuffle metric with stale-capacity reporting; stale-% column; debug-only table row counts; benchmarking paths, jemalloc, and a deploy image. The pg-vs-in-memory oracle suite stays green throughout, with new tests for the schema and correction paths.
define-null
force-pushed
the
defnull/net-681-multi-step-scheduling-3
branch
from
5f0a965 to
c22d18c
Compare
define-null
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this PR about?
Multi-step scheduling: correctness fixes, Postgres scale work, shared ring machinery.
This iteration takes the multi-step scheduler from test-scale to production-shaped data. Three tracks: fix two correctness bugs, push the per-cycle heavy lifting into Postgres, and consolidate the hash-ring code both schedulers duplicated. The reshuffle simulator gains the dataset operations and metrics to measure it.
Correctness
Postgres performance & scale
HashSet+ outerBTreeMapbecome a positionalVec, canonicalized only at the COPY boundary.Shared ring machinery: extracted into
src/rings.rs;WorkerRingCacherebuilds rings only when the worker id list changes, so a steady fleet reuses them across cycles.Per-dataset schema: a
schemastable pluschunks.schema_idlets files be derived from the schema instead of stored per-chunk; adds atables_presentbitmap and per-pipeline chunk types so the hot read decodes a slim row.Reshuffle simulator: step-triggered copy/replace sourced from live storage; ideal-vs-ideal shuffle metric with stale-capacity reporting; stale-% column; debug-only table row counts; benchmarking paths, jemalloc, and a deploy image.
The pg-vs-in-memory oracle suite stays green throughout, with new tests for the schema and correction paths.