feat(delphi): storage v2 P6b — --input-source read seam + implementation notes#2603
Open
jucor wants to merge 1 commit into
Open
feat(delphi): storage v2 P6b — --input-source read seam + implementation notes#2603jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
Collaborator
Author
|
Post-push self-review (Claude subagent): live paths verified regression-free; one important snapshot-path finding, now fixed and re-pushed:
Fix: Suite: 442 passed / 58 xfailed, goldens green, same 5 pre-existing DynamoDB-unavailable errors. |
jucor
force-pushed
the
jc/delphi-storage-v2-p6b
branch
from
July 6, 2026 22:32
68a0494 to
86710e9
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 6, 2026 22:52
3770aa7 to
2c2d396
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6b
branch
from
July 6, 2026 23:58
86710e9 to
704b041
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 6, 2026 23:58
2c2d396 to
00a4376
Compare
jucor
added a commit
that referenced
this pull request
Jul 7, 2026
…ycle + per-zid purge Implements the manifest half of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md §4.3, §6.1 M0→M1, §6.2): every pipeline job now leaves a v2 run manifest when v2 writes are enabled. Stage-group dual-writes follow (P7b-P7e). - delphi_storage/write_mode.py — DELPHI_WRITE_MODE=old|both|v2, the locked tri-state design: FAIL-LOUD when unset (a silently-defaulted writer would break the flip-back guarantee, §6.2 invariant 2). example.env and docker-compose.test.yml carry `old` (M0) so dev/test keep working; run_delphi exits 2 with a pointed message when unset. - delphi_storage/manifest.py — idempotent lifecycle helpers (ensure_run, mark_running, record_input_fingerprints, mark_completed, mark_failed) over the store's semantic ops, callable by BOTH the poller and run_delphi without coordination (complete_run was already idempotent/crash-healing). - scripts/job_poller.py — mirror_job_claimed / mirror_job_finished (module-level, testable without a JobProcessor): the OLD queue remains the single master (§6.2 invariant 3); v2 run rows are mirrors that become manifests as jobs execute. complete_job is the single funnel for every terminal path, so one hook covers success/failure/timeout/critical. Mirror errors are logged, NEVER raised — M1 requires the old serving path to survive a broken v2 store. Narrative job types deliberately not mirrored yet (P7d). - run_delphi.py — write-mode-driven behavior: old = today's; both/v2 = manifest + input snapshot + fingerprints merged into the manifest + completion/failure marks at every exit path. Failure policy: both → log-and-continue (old path keeps serving; unreplayability is visible as missing fingerprints); v2 or explicit --snapshot-inputs → abort. Replays (--input-source) never write manifests here (the replay harness, P12, creates its own fresh run). - delphi_storage/purge.py + scripts/delphi_purge.py — per-zid purge (list_runs + purge_job per run), completing the §9 GDPR path promised in P6a: snapshot/artifact partitions removed; manifests and latest pointers retained (they carry only config/counts/hashes — retention machinery removes them in P14). Tests (TDD; tests/test_write_mode_manifest.py, RED first): 23 tests — write-mode parsing/fail-loud/truth table, manifest helper idempotence and latest-flip, poller mirror for all modes/job types incl. the never-raise policy, run_delphi mode matrix (old/both/v2 × capture success/failure, explicit-flag abort, unset-mode fail-loud), per-zid purge isolation. Existing run_delphi-invoking suites updated to set the now-required mode. Full suite: 469 passed, 58 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed. Stack: P1 #2597 … P6b #2603, P7a this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
added a commit
that referenced
this pull request
Jul 7, 2026
…ycle + per-zid purge Implements the manifest half of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md §4.3, §6.1 M0→M1, §6.2): every pipeline job now leaves a v2 run manifest when v2 writes are enabled. Stage-group dual-writes follow (P7b-P7e). - delphi_storage/write_mode.py — DELPHI_WRITE_MODE=old|both|v2, the locked tri-state design: FAIL-LOUD when unset (a silently-defaulted writer would break the flip-back guarantee, §6.2 invariant 2). example.env and docker-compose.test.yml carry `old` (M0) so dev/test keep working; run_delphi exits 2 with a pointed message when unset. - delphi_storage/manifest.py — idempotent lifecycle helpers (ensure_run, mark_running, record_input_fingerprints, mark_completed, mark_failed) over the store's semantic ops, callable by BOTH the poller and run_delphi without coordination (complete_run was already idempotent/crash-healing). - scripts/job_poller.py — mirror_job_claimed / mirror_job_finished (module-level, testable without a JobProcessor): the OLD queue remains the single master (§6.2 invariant 3); v2 run rows are mirrors that become manifests as jobs execute. complete_job is the single funnel for every terminal path, so one hook covers success/failure/timeout/critical. Mirror errors are logged, NEVER raised — M1 requires the old serving path to survive a broken v2 store. Narrative job types deliberately not mirrored yet (P7d). - run_delphi.py — write-mode-driven behavior: old = today's; both/v2 = manifest + input snapshot + fingerprints merged into the manifest + completion/failure marks at every exit path. Failure policy: both → log-and-continue (old path keeps serving; unreplayability is visible as missing fingerprints); v2 or explicit --snapshot-inputs → abort. Replays (--input-source) never write manifests here (the replay harness, P12, creates its own fresh run). - delphi_storage/purge.py + scripts/delphi_purge.py — per-zid purge (list_runs + purge_job per run), completing the §9 GDPR path promised in P6a: snapshot/artifact partitions removed; manifests and latest pointers retained (they carry only config/counts/hashes — retention machinery removes them in P14). Tests (TDD; tests/test_write_mode_manifest.py, RED first): 23 tests — write-mode parsing/fail-loud/truth table, manifest helper idempotence and latest-flip, poller mirror for all modes/job types incl. the never-raise policy, run_delphi mode matrix (old/both/v2 × capture success/failure, explicit-flag abort, unset-mode fail-loud), per-zid purge isolation. Existing run_delphi-invoking suites updated to set the now-required mode. Full suite: 469 passed, 58 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed. Stack: P1 #2597 … P6b #2603, P7a this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
added a commit
that referenced
this pull request
Jul 7, 2026
…ycle + per-zid purge Implements the manifest half of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md §4.3, §6.1 M0→M1, §6.2): every pipeline job now leaves a v2 run manifest when v2 writes are enabled. Stage-group dual-writes follow (P7b-P7e). - delphi_storage/write_mode.py — DELPHI_WRITE_MODE=old|both|v2, the locked tri-state design: FAIL-LOUD when unset (a silently-defaulted writer would break the flip-back guarantee, §6.2 invariant 2). example.env and docker-compose.test.yml carry `old` (M0) so dev/test keep working; run_delphi exits 2 with a pointed message when unset. - delphi_storage/manifest.py — idempotent lifecycle helpers (ensure_run, mark_running, record_input_fingerprints, mark_completed, mark_failed) over the store's semantic ops, callable by BOTH the poller and run_delphi without coordination (complete_run was already idempotent/crash-healing). - scripts/job_poller.py — mirror_job_claimed / mirror_job_finished (module-level, testable without a JobProcessor): the OLD queue remains the single master (§6.2 invariant 3); v2 run rows are mirrors that become manifests as jobs execute. complete_job is the single funnel for every terminal path, so one hook covers success/failure/timeout/critical. Mirror errors are logged, NEVER raised — M1 requires the old serving path to survive a broken v2 store. Narrative job types deliberately not mirrored yet (P7d). - run_delphi.py — write-mode-driven behavior: old = today's; both/v2 = manifest + input snapshot + fingerprints merged into the manifest + completion/failure marks at every exit path. Failure policy: both → log-and-continue (old path keeps serving; unreplayability is visible as missing fingerprints); v2 or explicit --snapshot-inputs → abort. Replays (--input-source) never write manifests here (the replay harness, P12, creates its own fresh run). - delphi_storage/purge.py + scripts/delphi_purge.py — per-zid purge (list_runs + purge_job per run), completing the §9 GDPR path promised in P6a: snapshot/artifact partitions removed; manifests and latest pointers retained (they carry only config/counts/hashes — retention machinery removes them in P14). Tests (TDD; tests/test_write_mode_manifest.py, RED first): 23 tests — write-mode parsing/fail-loud/truth table, manifest helper idempotence and latest-flip, poller mirror for all modes/job types incl. the never-raise policy, run_delphi mode matrix (old/both/v2 × capture success/failure, explicit-flag abort, unset-mode fail-loud), per-zid purge isolation. Existing run_delphi-invoking suites updated to set the now-required mode. Full suite: 469 passed, 58 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed. Stack: P1 #2597 … P6b #2603, P7a this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion notes Implements the read half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.4, §5): stages can consume a recorded snapshot instead of live PG via --input-source=store://<job_id> — the seam the replay CLI (P12) drives. Approved via propose-then-wait (touches the vote-feeding loop in polismath/run_math_pipeline.py; the seam substitutes data sources only, never transformations). Seam mechanisms (in delphi_storage/inputs.py + minimal stage edits): - SnapshotReader — stage-shaped accessors over a snapshot: exact vote stream/batches, stage-1 comment/moderation rows, math_main with BOTH consumer semantics (env-less latest for 501/group_data; MATH_ENV-filtered for 801). - Stage 1: fetch_comments/fetch_moderation split into SQL fetch + SHARED row→dict transformations; the snapshot path routes through the same transformations, reproducing production behavior quirk-for-quirk — including the mod == '-1' int-vs-string comparisons that never fire (deliberately NOT fixed: golden invariance). - SnapshotPostgresClient — duck-typed read-only stand-in for the umap PostgresClient (conversation/comments/selections/votes_latest_unique with the PG-boundary sign flip); run_pipeline/501/801 run UNCHANGED against it; validates the requested zid against the snapshot's stamped zid. - GroupDataProcessor(math_main_override=...) and 801's _get_math_main_data branch for the raw-.query() math_main reads. - run_delphi.py forwards --input-source to the three PG-reading stages; --snapshot-inputs and --input-source are mutually exclusive (exit 2). Proof style: FEED EQUALITY (tests/test_input_source_seam.py, RED first) — live-vs-snapshot exact equality of every feed (comments dicts, moderation, each vote batch, stage-2 client structures, votes_latest_unique with flip, math_main per env), on a seeded scratch PG. Downstream code is untouched and seeded, so feed equality implies output equality; the golden suite stays the output gate. Also adds docs/STORAGE_V2_IMPLEMENTATION_NOTES.md: the decisions, invariants, and traps from P2 through P6b (cross-language contract, store semantics, DDL sync, job-id threading, snapshot fidelity, seam mechanisms, workspace/tooling gotchas, continuation checklist) — written so a future session, human or model, can continue without this session's context. Full suite: 442 passed, 57 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed. Stack: P1 #2597 … P6a #2602, P6b this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p6b
branch
from
July 10, 2026 11:40
704b041 to
b00eeab
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 10, 2026 11:40
00a4376 to
320db2c
Compare
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 / P6b of the Storage V2 effort (design in #2597, §4.4 + §5): the read half of input snapshots — stages consume a recorded snapshot instead of live PG via
--input-source=store://<job_id>, the seam the replay CLI (P12) will drive. Approved via propose-then-wait (it touches the vote-feeding loop inpolismath/run_math_pipeline.py); the seam substitutes data sources only, never transformations.Stacked on #2602 (
jc/delphi-storage-v2-p6a).Seam mechanisms (three, by preference — documented in the notes doc)
fetch_comments/fetch_moderationwere split into SQL fetch + shared row→dict transformations; the snapshot path (comments_from_snapshot/moderation_from_snapshot) feeds stage-shaped rows through the same transformations. This reproduces production behavior quirk-for-quirk — including themod == '-1'int-vs-string comparisons that never fire (deliberately not fixed: they're production math inputs; golden invariance).SnapshotPostgresClient(indelphi_storage/inputs.py) stands in for the umapPostgresClientreads — conversation, comments, selections, andvotes_latest_unique(derived from the raw stream, with the PG-boundary sign flip).run_pipeline.py, 501 and 801 run unchanged against it; it validates the requested zid against the snapshot's stamped zid..query()SQL:GroupDataProcessor(math_main_override=…)(env-less-latest semantics, 501) and a branch in 801's_get_math_main_data(MATH_ENV-filtered semantics) — each preserving the exact semantics of the SQL it replaces.run_delphi.pyforwards--input-sourceto the three PG-reading stages (502/700 read DynamoDB; reset has no inputs);--snapshot-inputsand--input-sourceare mutually exclusive (exit 2).Proof style: feed equality
tests/test_input_source_seam.py(TDD, RED first — 26 tests) asserts live-vs-snapshot exact equality of every feed on a seeded scratch PG: stage-1 comments dicts, moderation dict, every vote batch at production batch slicing, stage-2 client structures,votes_latest_uniquewith the flip,math_mainunder both consumers' env semantics. The downstream code is untouched and seeded, so feed equality ⟹ output equality; the golden suite remains the output-invariance gate (ran and passed). A full pipeline-run comparison would only add DynamoDB/EVōC noise without adding proof.Implementation notes for future sessions
Also adds
delphi/docs/STORAGE_V2_IMPLEMENTATION_NOTES.md— the decisions, invariants and traps from P2 through P6b (hard constraints incl. quirk parity and vote-order load-bearing-ness; the cross-language wire contract; store semantics incl. generation-tagged chunking and the commit-point ordering; DDL sync mechanics; job-id threading; snapshot fidelity incl. the tie caveat; the seam mechanisms above; workspace/tooling gotchas; a P7+ continuation checklist). Written explicitly so work can continue in a future session without this session's context — per-PR review comments complement it.Testing
Stack
DELPHI_WRITE_MODE), per the continuation checklist in the notes doc🤖 Generated with Claude Code