feat(delphi): storage v2 P6a — per-run input snapshots (capture side) + purge tool#2602
Open
jucor wants to merge 1 commit into
Open
feat(delphi): storage v2 P6a — per-run input snapshots (capture side) + purge tool#2602jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
Collaborator
Author
|
Post-push self-review (Claude subagent) found four issues, now addressed and re-pushed:
Suite: 430 passed / 58 xfailed, goldens green, same 5 pre-existing DynamoDB-unavailable errors. |
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 6, 2026 16:08
b809c80 to
2b8ddfe
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p5
branch
from
July 6, 2026 18:48
95099f9 to
abac670
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 6, 2026 18:48
2b8ddfe to
3770aa7
Compare
jucor
added a commit
that referenced
this pull request
Jul 6, 2026
…ion notes Implements the read half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.4, §5): stages can consume a recorded snapshot instead of live PG via --input-source=store://<job_id> — the seam the replay CLI (P12) drives. Approved via propose-then-wait (touches the vote-feeding loop in polismath/run_math_pipeline.py; the seam substitutes data sources only, never transformations). Seam mechanisms (in delphi_storage/inputs.py + minimal stage edits): - SnapshotReader — stage-shaped accessors over a snapshot: exact vote stream/batches, stage-1 comment/moderation rows, math_main with BOTH consumer semantics (env-less latest for 501/group_data; MATH_ENV-filtered for 801). - Stage 1: fetch_comments/fetch_moderation split into SQL fetch + SHARED row→dict transformations; the snapshot path routes through the same transformations, reproducing production behavior quirk-for-quirk — including the mod == '-1' int-vs-string comparisons that never fire (deliberately NOT fixed: golden invariance). - SnapshotPostgresClient — duck-typed read-only stand-in for the umap PostgresClient (conversation/comments/selections/votes_latest_unique with the PG-boundary sign flip); run_pipeline/501/801 run UNCHANGED against it; validates the requested zid against the snapshot's stamped zid. - GroupDataProcessor(math_main_override=...) and 801's _get_math_main_data branch for the raw-.query() math_main reads. - run_delphi.py forwards --input-source to the three PG-reading stages; --snapshot-inputs and --input-source are mutually exclusive (exit 2). Proof style: FEED EQUALITY (tests/test_input_source_seam.py, RED first) — live-vs-snapshot exact equality of every feed (comments dicts, moderation, each vote batch, stage-2 client structures, votes_latest_unique with flip, math_main per env), on a seeded scratch PG. Downstream code is untouched and seeded, so feed equality implies output equality; the golden suite stays the output gate. Also adds docs/STORAGE_V2_IMPLEMENTATION_NOTES.md: the decisions, invariants, and traps from P2 through P6b (cross-language contract, store semantics, DDL sync, job-id threading, snapshot fidelity, seam mechanisms, workspace/tooling gotchas, continuation checklist) — written so a future session, human or model, can continue without this session's context. Full suite: 442 passed, 57 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed. Stack: P1 #2597 … P6a #2602, P6b this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p5
branch
from
July 6, 2026 22:52
abac670 to
df5baee
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 6, 2026 22:52
3770aa7 to
2c2d396
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the “capture side” of Storage V2 input snapshots for Delphi pipeline runs, plus a per-job purge utility, to support reproducible runs by recording exactly what the pipeline read from source Postgres at job start.
Changes:
- Introduces
delphi_storage.inputs.capture_run_inputs(...)to snapshot six input kinds intorun_inputswith codec envelopes + fingerprints. - Adds
delphi_storage.purge.purge_job(...)to delete all generic-entity partitions for a job (GDPR/retention foundation). - Hooks
run_delphi.pyto optionally snapshot inputs before running any stages, and imports shared vote SQL constants into stage 1 to prevent drift.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| delphi/delphi_storage/inputs.py | Implements snapshot capture + fingerprints and exposes shared vote SQL constants. |
| delphi/delphi_storage/purge.py | Adds a per-job purge helper that deletes all generic-entity partitions for a job_id. |
| delphi/run_delphi.py | Adds --snapshot-inputs / env flag hook to capture inputs pre-stage and abort on capture failure. |
| delphi/polismath/run_math_pipeline.py | Imports vote SQL constants from snapshot module and uses them for stage 1 vote reads. |
| delphi/tests/test_input_snapshots.py | Adds Postgres-backed tests covering snapshot fidelity, SQL parity, purge behavior, and run_delphi hook behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+62
to
+69
| return psycopg2.connect( | ||
| host=os.environ.get("DATABASE_HOST", "localhost"), | ||
| port=os.environ.get("DATABASE_PORT", "5432"), | ||
| dbname=os.environ.get("DATABASE_NAME", "polis-dev"), | ||
| user=os.environ.get("DATABASE_USER", "postgres"), | ||
| password=os.environ.get("DATABASE_PASSWORD", ""), | ||
| connect_timeout=timeout, | ||
| ) |
| if _IN_GHA: | ||
| raise | ||
| pytest.skip(f"PostgreSQL unavailable: {e}") | ||
| scratch_url = base_url.rsplit("/", 1)[0] + f"/{dbname}" |
Comment on lines
+138
to
+144
| vote_rows: list = [] | ||
| for offset in range(0, max(total_votes, 1), batch_size): | ||
| cursor = conn.cursor() | ||
| cursor.execute(VOTES_BATCH_SQL, (zid, batch_size, offset)) | ||
| vote_rows.extend(cursor.fetchall()) | ||
| cursor.close() | ||
| payload = _snapshot_payload(["created", "tid", "pid", "vote"], vote_rows) |
jucor
added a commit
that referenced
this pull request
Jul 6, 2026
…ion notes Implements the read half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.4, §5): stages can consume a recorded snapshot instead of live PG via --input-source=store://<job_id> — the seam the replay CLI (P12) drives. Approved via propose-then-wait (touches the vote-feeding loop in polismath/run_math_pipeline.py; the seam substitutes data sources only, never transformations). Seam mechanisms (in delphi_storage/inputs.py + minimal stage edits): - SnapshotReader — stage-shaped accessors over a snapshot: exact vote stream/batches, stage-1 comment/moderation rows, math_main with BOTH consumer semantics (env-less latest for 501/group_data; MATH_ENV-filtered for 801). - Stage 1: fetch_comments/fetch_moderation split into SQL fetch + SHARED row→dict transformations; the snapshot path routes through the same transformations, reproducing production behavior quirk-for-quirk — including the mod == '-1' int-vs-string comparisons that never fire (deliberately NOT fixed: golden invariance). - SnapshotPostgresClient — duck-typed read-only stand-in for the umap PostgresClient (conversation/comments/selections/votes_latest_unique with the PG-boundary sign flip); run_pipeline/501/801 run UNCHANGED against it; validates the requested zid against the snapshot's stamped zid. - GroupDataProcessor(math_main_override=...) and 801's _get_math_main_data branch for the raw-.query() math_main reads. - run_delphi.py forwards --input-source to the three PG-reading stages; --snapshot-inputs and --input-source are mutually exclusive (exit 2). Proof style: FEED EQUALITY (tests/test_input_source_seam.py, RED first) — live-vs-snapshot exact equality of every feed (comments dicts, moderation, each vote batch, stage-2 client structures, votes_latest_unique with flip, math_main per env), on a seeded scratch PG. Downstream code is untouched and seeded, so feed equality implies output equality; the golden suite stays the output gate. Also adds docs/STORAGE_V2_IMPLEMENTATION_NOTES.md: the decisions, invariants, and traps from P2 through P6b (cross-language contract, store semantics, DDL sync, job-id threading, snapshot fidelity, seam mechanisms, workspace/tooling gotchas, continuation checklist) — written so a future session, human or model, can continue without this session's context. Full suite: 442 passed, 57 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed. Stack: P1 #2597 … P6a #2602, P6b this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p5
branch
from
July 6, 2026 23:58
df5baee to
90dc74d
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 6, 2026 23:58
2c2d396 to
00a4376
Compare
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p5
branch
from
July 10, 2026 11:40
90dc74d to
cf3898c
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p6a
branch
from
July 10, 2026 11:40
00a4376 to
320db2c
Compare
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 / P6a of the Storage V2 effort (design in #2597, §4.2 entity 2 + §4.4): the capture half of input snapshots — reproducibility starts with recording exactly what the pipeline read. The read seam (
--input-source, P6b) follows as its own PR, per the design's deliberate 6a/6b isolation of the riskiest step.Stacked on #2601 (
jc/delphi-storage-v2-p5).What this adds
delphi_storage/inputs.py— snapshot capturecapture_run_inputs(store, job_id, zid, rid)writes the six snapshot kinds asrun_inputsitems (codec envelopes; votes always zstd-compressed) and returns per-kind fingerprints (sha256 of the canonical uncompressed payload,row_count, max votecreated) — exactly what P7's manifest records.The fidelity decisions, each pinned by a test:
ORDER BY created, raw PG signs — recon confirmed the production math path never flips the sign (fetch_votes()'s flip is dead code) — superseded votes included, order preserved (the math KMeans init deliberately depends on vote-encounter order, seeINVESTIGATION_K_DIVERGENCE.md).run_math_pipeline.pynow importsVOTES_COUNT_SQL/VOTES_BATCH_SQLfromdelphi_storage.inputs(strings unchanged — the golden suite and the e2e SQL-pin test stay green), so capture and stage 1 can never diverge.mod = -1) and banned participants — moderation filtering is pipeline behavior, not snapshot behavior (the snapshot must let a replay reproduce whatever filtering the code-at-replay-time does).clojure_math_maincaptured for everymath_env— 501 reads the env-less latest while 801 filters byMATH_ENVwith a different default ('prod' vs 'dev'); snapshotting all envs keeps both replayable.derive_votes_latest_unique()reproduces PG's RULE-maintainedvotes_latest_uniquefrom the stream (last vote per(pid, tid)in stream order) — the in-pipeline derivation §4.4 calls for, tested against a seeded PG table that mimics the RULE (including a re-vote).delphi_storage/purge.py— the §9 GDPR pathpurge_job()deletes every generic-entity partition for a job — snapshots copy votes/comments outside the source PG, so deletion needs this from day one.run_delphi.pyhook--snapshot-inputs(orDELPHI_SNAPSHOT_INPUTS=1) captures before any stage runs and aborts the run on capture failure — a run without recorded inputs defeats the point when enabled. Off by default until P7'sDELPHI_WRITE_MODEsubsumes it (M1 in the migration plan).Testing (TDD — 16 tests written first, RED on missing module)
votes_latest_unique;run_math_pipeline.VOTES_BATCH_SQL is delphi_storage.inputs.VOTES_BATCH_SQL);Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged). Golden snapshots ran and passed — math outputs untouched.
Stack
--input-source=store://<job_id>seam), votes_latest_unique derivation wired to consumers, golden invariance again as the gate🤖 Generated with Claude Code