feat(delphi): storage v2 P4 — deployable DDL (Delphi2 Dynamo tables + PG migration 000019)#2600
Open
jucor wants to merge 1 commit into
Open
feat(delphi): storage v2 P4 — deployable DDL (Delphi2 Dynamo tables + PG migration 000019)#2600jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
jucor
force-pushed
the
jc/delphi-storage-v2-p3
branch
from
57be28e to
9ee292b
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p4
branch
from
3426ccb to
5eeaaa7
Compare
Collaborator
Author
|
Post-push self-review (Claude subagent) found three issues, now fixed and re-pushed:
All 7 DDL tests green against DynamoDB local + dockerized PG. |
jucor
force-pushed
the
jc/delphi-storage-v2-p4
branch
2 times, most recently
from
bc792db to
5a5e0d4
Compare
jucor
added a commit
that referenced
this pull request
Jul 6, 2026
… pipeline entry points Implements Stack 2 / P5 of docs/STORAGE_V2_DESIGN.md (§4.4): the pipeline job id travels ON THE COMMAND LINE instead of the env-var-only propagation that reached 2 of 18 tables (§3.2). - delphi_storage/job_id.py — resolve_job_id: explicit CLI value > DELPHI_JOB_ID env (transition fallback, removed in a later phase) > auto local-<uuid4> for dev runs. - run_delphi.py — gains --job-id, resolves once, exports DELPHI_JOB_ID for un-migrated env readers (same id as the command lines), and appends --job-id=<id> to all six stage subprocesses (reset, math, umap 500s, 501, 502, 700 per layer). - scripts/job_poller.py — command construction extracted into module-level build_job_command() (now unit-testable); FULL_PIPELINE and CREATE_NARRATIVE_BATCH commands carry --job-id=<queue job_id>; the DELPHI_JOB_ID env export stays for the transition window. 803's existing --job-id (the batch-tracking id, a distinct concept) is unchanged. - Stage entry points all accept --job-id: reset_conversation (CLI moved into cli(), main() stays the worker), run_math_pipeline (arg + log only — no math-logic change), run_pipeline (threaded into process_conversation, replacing the buried env read; auto prefix changes pipeline_run_<uuid> → local-<uuid> for bare dev runs), 501, 502 (arg + log, used from P7), 700 (threaded down to the S3-key builder; 'unknown' fallback preserved), 801 (arg > env, NO auto-generation — narrative section keys embed the id, so a missing one must keep failing loudly downstream). Tests (TDD; tests/test_job_id_threading.py, RED first): - resolve_job_id precedence; run_delphi threads one shared id to all six stage commands (explicit, auto, and env-fallback variants); - every stage entry point's --help advertises --job-id; - process_conversation exposes the job_id parameter; - poller commands for all three job types (FULL_PIPELINE + 801 carry the queue job id; 803 keeps batch_job_id semantics). Full suite: 424 passed, 30 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — math outputs untouched. Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
added a commit
that referenced
this pull request
Jul 6, 2026
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p3
branch
from
9ee292b to
5edc451
Compare
jucor
added a commit
that referenced
this pull request
Jul 6, 2026
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p4
branch
from
5a5e0d4 to
5f9fbd5
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p4
branch
from
5f9fbd5 to
efc507b
Compare
jucor
added a commit
that referenced
this pull request
Jul 6, 2026
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds deployable DDL artifacts for Delphi Storage V2 (PostgreSQL migration + DynamoDB table creation) and verifies they stay in lockstep with the backend-defined schema via sync + conformance tests, with CI wiring to run those tests in the existing Docker-based pipeline.
Changes:
- Add PostgreSQL migration
000019to create thedelphischema and its six V2 entities + indexes. - Extend
delphi/create_dynamodb_tables.pyto always create theDelphi2_*DynamoDB tables (with an inlined schema copy, drift-checked by tests). - Add DDL drift + deployed-artifact conformance tests, and update CI/docker-compose test env to support SQLAlchemy-safe Postgres URLs and migration staging.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| server/postgres/migrations/000019_create_delphi_storage.sql | New migration creating the V2 delphi schema tables and indexes. |
| delphi/create_dynamodb_tables.py | Adds Delphi2 table schemas + creation helper and wires it into the standard table provisioning flow. |
| delphi/tests/test_delphi_storage_ddl.py | New tests to enforce DDL drift-sync and validate deployed DDL artifacts via conformance runs. |
| docker-compose.test.yml | Passes DELPHI_STORAGE_PG_URL/DATABASE_URL into the Delphi test container for PG-backend tests. |
| .github/workflows/python-ci.yml | Derives DELPHI_STORAGE_PG_URL in CI and stages server migrations into the Delphi test container. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+37
to
+38
| def _normalize_sql(sql: str) -> str: | ||
| return re.sub(r"\s+", " ", sql).strip().lower() |
Comment on lines
+136
to
+137
| store = PostgresDelphiStore(url=scratch_url, schema="delphi") | ||
| try: |
Comment on lines
+184
to
+186
| store = DynamoDelphiStore( | ||
| table_prefix="Delphi2_", endpoint_url=endpoint, region="us-east-1" | ||
| ) |
jucor
force-pushed
the
jc/delphi-storage-v2-p3
branch
from
5edc451 to
a8ff600
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p4
branch
from
efc507b to
68aa10e
Compare
… PG migration 000019)
Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment
artifacts for the V2 schema, pinned to the backend definitions.
- create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables()
wired into create_tables() (always created, like the job queue). The
schemas are an INLINED copy because the script runs standalone in the
dynamodb-init container (bare python + boto3, cannot import
delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts
from delphi_storage.backends.dynamodb.table_schemas("Delphi2_").
- server/postgres/migrations/000019_create_delphi_storage.sql: schema
`delphi` with runs / latest / run_inputs / artifacts / topic_moderation /
collective_statements, mirroring
delphi_storage.backends.postgres.schema_ddl("delphi") statement for
statement (drift-tested the same way).
- tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the
ensure_* test helpers) yield conformant stores: the migration applied by
raw SQL to a scratch database passes the full conformance suite with
ensure_schema off, and script-created Delphi2_* tables pass it with
ensure_tables off. Service-gated tests follow the skip-locally /
fail-in-GitHub-Actions convention.
- CI wiring (.github/workflows/python-ci.yml, docker-compose.test.yml) so the
fail-in-GHA tests above actually run instead of erroring: the delphi `test`
job now (a) stages server/postgres/migrations at /server/postgres/migrations
inside the container — only delphi/ is baked into the image, but the
migration-sync tests resolve the file at parents[2]/server/... which is
/server/... from /app/tests; and (b) provides DELPHI_STORAGE_PG_URL with the
postgresql:// scheme (SQLAlchemy 2.0 rejects the legacy postgres:// alias
that DATABASE_URL carries, which otherwise raised NoSuchModuleError at
engine creation), derived from the committed DATABASE_URL so credentials
stay single-sourced. Without this the whole delphi test job failed at
collection first (delphi_storage not on the image — fixed in P2) and then on
these PG/DDL tests.
Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG
all 6 DDL tests pass.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1
(Foundation); Stack 2 (provenance plumbing, P5-P8) follows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p3
branch
from
a8ff600 to
9ff55da
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p4
branch
from
68aa10e to
926e37b
Compare
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 1 / P4 of the Storage V2 effort (design in #2597, §4.2 "DDL" + §7): the deployable DDL for the V2 schema, pinned to the backend definitions from #2598/#2599. Completes Stack 1 (Foundation).
Stacked on #2599 (
jc/delphi-storage-v2-p3).What this adds
DynamoDB —
delphi/create_dynamodb_tables.pyDELPHI2_TABLE_SCHEMAS+create_delphi2_tables(), wired intocreate_tables()so theDelphi2_*tables are always created alongside the existing ones (dev, CIdynamodb-init, and instance provisioning all flow through this script).dynamodb-initcontainer (barepython:3.11-slim+ boto3 — it cannot importdelphi_storage). The sync testtests/test_delphi_storage_ddl.py::TestDynamoSchemaSyncfails CI if the copy ever drifts fromdelphi_storage.backends.dynamodb.table_schemas("Delphi2_")— the backend stays the single source of truth.PostgreSQL —
server/postgres/migrations/000019_create_delphi_storage.sqldelphiwith the six entities (runs,latest,run_inputs,artifacts,topic_moderation,collective_statements),COLLATE "C"sort keys, the partial claim index, and the zid/rid list indexes — mirroringdelphi_storage.backends.postgres.schema_ddl("delphi")statement for statement (drift-tested the same way).bin/run-migrations.sh,bin/db-reset.js); first migration to use a schema, per the design's decision to namespace V2 underdelphi.Proof the deployed artifacts work (not just the test helpers)
tests/test_delphi_storage_ddl.py::TestDeployedStoresPassConformance:ensure_schemaoff;Delphi2_*tables viacreate_delphi2_tables()against DynamoDB local and runs the full conformance suite withensure_tablesoff.Both follow the skip-locally / fail-in-GitHub-Actions convention (both services are up in CI).
Testing
:8002) + dockerized PG 16.python3 -m py_compile create_dynamodb_tables.py— the script stays stdlib+boto3-only.Stack
delphi_storage/)🤖 Generated with Claude Code