Skip to content

feat(delphi): storage v2 P7a — DELPHI_WRITE_MODE + run-manifest lifecycle + per-zid purge#2604

Open
jucor wants to merge 1 commit into
jc/delphi-storage-v2-p6bfrom
jc/delphi-storage-v2-p7a
Open

feat(delphi): storage v2 P7a — DELPHI_WRITE_MODE + run-manifest lifecycle + per-zid purge#2604
jucor wants to merge 1 commit into
jc/delphi-storage-v2-p6bfrom
jc/delphi-storage-v2-p7a

Conversation

@jucor

@jucor jucor commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Stack 2 / P7a of the Storage V2 effort (design in #2597, §4.3 + §6.1 M0→M1 + §6.2): the manifest half of P7 — every pipeline job leaves a v2 run manifest when v2 writes are enabled. The stage-group dual-writes (math, umap 500s, 501/502/700s, 801/803) follow as P7b–P7e, each with its verify_dual_write parity test.

Stacked on #2603 (jc/delphi-storage-v2-p6b).

What this adds

DELPHI_WRITE_MODE (delphi_storage/write_mode.py)

The locked tri-state from the design: old (M0, today's behavior) / both (the whole M1–M5 window — what makes the M4 flip-back lossless per §6.2 invariant 2) / v2 (M6+/fresh deployments). Fail-loud when unset — a silently-defaulted writer would undermine the flip-back guarantee undetected. example.env and docker-compose.test.yml carry old, so dev/test environments keep working; run_delphi.py exits 2 with a pointed message when unset, the poller mirror logs per-job instead of taking the worker down.

Run-manifest lifecycle (delphi_storage/manifest.py)

Idempotent helpers (ensure_run, mark_running, record_input_fingerprints, mark_completed, mark_failed) over the store's semantic ops. Idempotence is the coordination story: the poller AND run_delphi.py can both drive the same run without stepping on each other (complete_run was already idempotent/crash-healing from P2).

Poller mirror (scripts/job_poller.py)

mirror_job_claimed / mirror_job_finished — module-level (testable without a JobProcessor). The old queue remains the single master (§6.2 invariant 3): v2 run rows are mirrors that become manifests as jobs execute; real v2 claim semantics activate at the enqueue flip (P11). complete_job is the single funnel for every terminal path (success/failure/timeout/critical), so one hook covers them all. Mirror errors are logged, never raised — M1 requires the old serving path to survive a broken v2 store; divergence is caught by the coverage/verify tooling. Narrative job types are deliberately not mirrored yet (P7d).

run_delphi.py write-mode behavior

  • old: exactly today's behavior (explicit --snapshot-inputs still works, with its abort-on-failure semantics).
  • both/v2: manifest created + marked RUNNING, inputs snapshotted, fingerprints merged into the manifest, and COMPLETED/FAILED marked at every exit path.
  • Failure policy: both → log-and-continue (the old path keeps serving; an unreplayable run is visible as missing fingerprints); v2 → abort (there is no old copy to lean on).
  • Replays (--input-source) never write manifests here — the replay harness (P12) creates its own fresh run with replay_of.

Per-zid purge (completes the §9 GDPR path promised in #2602)

purge_zid (delphi_storage/purge.py) + scripts/delphi_purge.py --zid N | --job-id X: enumerates a conversation's runs via the now-existing manifests and removes every snapshot/artifact partition. Manifests and latest pointers are retained — they carry only config/counts/hashes, never vote/comment content; the retention machinery (P14) removes them.

Testing (TDD — 23 tests written first, RED on missing modules)

tests/test_write_mode_manifest.py: write-mode parsing / fail-loud / truth table; manifest-helper idempotence incl. the latest-pointer flip staying at seq 1 on double completion; the poller mirror across modes and job types incl. the never-raise policy; the run_delphi mode matrix (old/both/v2 × capture success/failure, explicit-flag abort, unset-mode fail-loud with zero stages run); per-zid purge isolation. Existing run_delphi-invoking suites updated for the now-required mode env.

Full suite: 469 passed, 58 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged). Golden snapshots ran and passed.

Stack

🤖 Generated with Claude Code

@jucor
jucor force-pushed the jc/delphi-storage-v2-p7a branch 2 times, most recently from d7c3ea3 to 14c9f43 Compare July 7, 2026 00:25
@jucor

jucor commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Post-push self-review (Claude subagent) — one critical finding, fixed and pushed:

The main docker-compose.yml never passed DELPHI_WRITE_MODE into the delphi container. Compose only forwards variables explicitly listed under a service's environment: — so despite example.env carrying DELPHI_WRITE_MODE=old, the container never saw it, and with the (by-design) fail-loud resolution every FULL_PIPELINE job under make start would have exited 2 immediately on merge — silent at the poller level (its mirror never raises), visible only as every job failing. Fixed: - DELPHI_WRITE_MODE=${DELPHI_WRITE_MODE:-old} added to the delphi service, mirroring docker-compose.test.yml. Deploy note for ops: the real AWS task definition for the delphi container needs the same variable set explicitly at M0 rollout (outside this repo).

Reviewer also confirmed: the alphanumeric report-id handling is correct in the pushed PR (recorded verbatim in config_requested, never int()-coerced — the earlier coercion was caught and fixed before review completed); mirror-before-legacy-write in complete_job is the documented accepted tradeoff; no module-level delphi_storage imports (no poller startup cost); AWAITING release_lock path correctly un-mirrored; test env hygiene clean.

@jucor jucor mentioned this pull request Jul 7, 2026
Open
jucor added a commit that referenced this pull request Jul 7, 2026
@jucor @claude
feat(delphi): storage v2 P7b — math stage dual-write + verify_dual_wr…
50f30e3 
…ite parity checker

Implements the math stage-group of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md
§4.2 artifacts, §6.1 M1, §7): the math results become v2 artifacts alongside
the legacy Delphi_* tables, gated by DELPHI_WRITE_MODE.

- polismath/database/v2_artifacts.py — build_math_artifacts derives the v2
  payloads from the SAME objects the legacy writer consumes
  (conv.to_dynamo_dict() output, conv.proj, conv.group_clusters), converting
  Decimals back to floats by value. Artifact keys per design §4.2:
  math#pca (incl. the FULL consensus dict — legacy keeps only the agree
  list), math#kmeans, math#repness, math#routing#<chunk>,
  math#projections#<chunk> (logical chunks of 5000 participants; 18k+
  happen in production — layered over the backend's byte-level
  auto-chunking). LEGACY QUIRKS REPRODUCED DELIBERATELY, each pinned by a
  test: routing priority/consensus default to 0 (never omitted), and the
  participant→group map is RAW-keyed exactly like the legacy writer — a
  str/int key mismatch yields group_id -1 on BOTH sides.
- write_math_artifacts stores codec envelopes under pk=job_id, records
  math_tick_legacy on the manifest and appends a log line (best-effort for
  standalone runs without a manifest row).
- run_math_pipeline.py — the write block becomes persist_math_results(conv,
  job_id, write_mode): old → legacy export only (unchanged behavior,
  extracted verbatim into _export_to_legacy_dynamo); both → legacy AND v2,
  v2 failures logged never raised (old path keeps serving); v2 → v2 only,
  failures exit 1 so the poller marks the run FAILED.
- scripts/verify_dual_write.py — the M1 parity gate: reads the legacy
  tables (at latest_math_tick) and the decoded v2 artifacts back
  INDEPENDENTLY (zero shared code with the writer) and compares numerically
  (Decimal vs float by value). CLI: --job-id/--zid/--endpoint-url, exit
  0/1. During development it caught three real divergences in the first
  builder draft (consensus agree-list only, routing defaults, raw-keyed
  group map) — exactly the failure mode it exists to catch.

Tests (TDD; tests/test_math_dual_write.py, RED first): 12 tests — builder
payloads/joins/chunking/quirks, envelope round-trip + manifest bookkeeping,
mode gating incl. failure policies, and an end-to-end parity test against
real DynamoDB local (legacy write via the REAL DynamoDBClient, then
verify passes; corrupt one v2 value, verify catches it). Skip-locally /
fail-in-CI convention.

Full suite: 480 passed, 59 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched (write-side only).

Stack: P1 #2597 … P7a #2604, P7b this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor added a commit that referenced this pull request Jul 7, 2026
@jucor @claude
feat(delphi): storage v2 P7b — math stage dual-write + verify_dual_wr…
d950db4 
…ite parity checker

Implements the math stage-group of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md
§4.2 artifacts, §6.1 M1, §7): the math results become v2 artifacts alongside
the legacy Delphi_* tables, gated by DELPHI_WRITE_MODE.

- polismath/database/v2_artifacts.py — build_math_artifacts derives the v2
  payloads from the SAME objects the legacy writer consumes
  (conv.to_dynamo_dict() output, conv.proj, conv.group_clusters), converting
  Decimals back to floats by value. Artifact keys per design §4.2:
  math#pca (incl. the FULL consensus dict — legacy keeps only the agree
  list), math#kmeans, math#repness, math#routing#<chunk>,
  math#projections#<chunk> (logical chunks of 5000 participants; 18k+
  happen in production — layered over the backend's byte-level
  auto-chunking). LEGACY QUIRKS REPRODUCED DELIBERATELY, each pinned by a
  test: routing priority/consensus default to 0 (never omitted), and the
  participant→group map is RAW-keyed exactly like the legacy writer — a
  str/int key mismatch yields group_id -1 on BOTH sides.
- write_math_artifacts stores codec envelopes under pk=job_id, records
  math_tick_legacy on the manifest and appends a log line (best-effort for
  standalone runs without a manifest row).
- run_math_pipeline.py — the write block becomes persist_math_results(conv,
  job_id, write_mode): old → legacy export only (unchanged behavior,
  extracted verbatim into _export_to_legacy_dynamo); both → legacy AND v2,
  v2 failures logged never raised (old path keeps serving); v2 → v2 only,
  failures exit 1 so the poller marks the run FAILED.
- scripts/verify_dual_write.py — the M1 parity gate: reads the legacy
  tables (at latest_math_tick) and the decoded v2 artifacts back
  INDEPENDENTLY (zero shared code with the writer) and compares numerically
  (Decimal vs float by value). CLI: --job-id/--zid/--endpoint-url, exit
  0/1. During development it caught three real divergences in the first
  builder draft (consensus agree-list only, routing defaults, raw-keyed
  group map) — exactly the failure mode it exists to catch.

Tests (TDD; tests/test_math_dual_write.py, RED first): 12 tests — builder
payloads/joins/chunking/quirks, envelope round-trip + manifest bookkeeping,
mode gating incl. failure policies, and an end-to-end parity test against
real DynamoDB local (legacy write via the REAL DynamoDBClient, then
verify passes; corrupt one v2 value, verify catches it). Skip-locally /
fail-in-CI convention.

Full suite: 480 passed, 59 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched (write-side only).

Stack: P1 #2597 … P7a #2604, P7b this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor added a commit that referenced this pull request Jul 7, 2026
@jucor @claude
feat(delphi): storage v2 P7b — math stage dual-write + verify_dual_wr…
dcab2b1 
…ite parity checker

Implements the math stage-group of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md
§4.2 artifacts, §6.1 M1, §7): the math results become v2 artifacts alongside
the legacy Delphi_* tables, gated by DELPHI_WRITE_MODE.

- polismath/database/v2_artifacts.py — build_math_artifacts derives the v2
  payloads from the SAME objects the legacy writer consumes
  (conv.to_dynamo_dict() output, conv.proj, conv.group_clusters), converting
  Decimals back to floats by value. Artifact keys per design §4.2:
  math#pca (incl. the FULL consensus dict — legacy keeps only the agree
  list), math#kmeans, math#repness, math#routing#<chunk>,
  math#projections#<chunk> (logical chunks of 5000 participants; 18k+
  happen in production — layered over the backend's byte-level
  auto-chunking). LEGACY QUIRKS REPRODUCED DELIBERATELY, each pinned by a
  test: routing priority/consensus default to 0 (never omitted), and the
  participant→group map is RAW-keyed exactly like the legacy writer — a
  str/int key mismatch yields group_id -1 on BOTH sides.
- write_math_artifacts stores codec envelopes under pk=job_id, records
  math_tick_legacy on the manifest and appends a log line (best-effort for
  standalone runs without a manifest row).
- run_math_pipeline.py — the write block becomes persist_math_results(conv,
  job_id, write_mode): old → legacy export only (unchanged behavior,
  extracted verbatim into _export_to_legacy_dynamo); both → legacy AND v2,
  v2 failures logged never raised (old path keeps serving); v2 → v2 only,
  failures exit 1 so the poller marks the run FAILED.
- scripts/verify_dual_write.py — the M1 parity gate: reads the legacy
  tables (at latest_math_tick) and the decoded v2 artifacts back
  INDEPENDENTLY (zero shared code with the writer) and compares numerically
  (Decimal vs float by value). CLI: --job-id/--zid/--endpoint-url, exit
  0/1. During development it caught three real divergences in the first
  builder draft (consensus agree-list only, routing defaults, raw-keyed
  group map) — exactly the failure mode it exists to catch.

Tests (TDD; tests/test_math_dual_write.py, RED first): 12 tests — builder
payloads/joins/chunking/quirks, envelope round-trip + manifest bookkeeping,
mode gating incl. failure policies, and an end-to-end parity test against
real DynamoDB local (legacy write via the REAL DynamoDBClient, then
verify passes; corrupt one v2 value, verify catches it). Skip-locally /
fail-in-CI convention.

Full suite: 480 passed, 59 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched (write-side only).

Stack: P1 #2597 … P7a #2604, P7b this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jucor jucor mentioned this pull request Jul 7, 2026
Open
@jucor @claude
feat(delphi): storage v2 P7a — DELPHI_WRITE_MODE + run-manifest lifec…
c3c973f 
…ycle + per-zid purge

Implements the manifest half of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md
§4.3, §6.1 M0→M1, §6.2): every pipeline job now leaves a v2 run manifest
when v2 writes are enabled. Stage-group dual-writes follow (P7b-P7e).

- delphi_storage/write_mode.py — DELPHI_WRITE_MODE=old|both|v2, the locked
  tri-state design: FAIL-LOUD when unset (a silently-defaulted writer would
  break the flip-back guarantee, §6.2 invariant 2). example.env and
  docker-compose.test.yml carry `old` (M0) so dev/test keep working;
  run_delphi exits 2 with a pointed message when unset.
- delphi_storage/manifest.py — idempotent lifecycle helpers (ensure_run,
  mark_running, record_input_fingerprints, mark_completed, mark_failed)
  over the store's semantic ops, callable by BOTH the poller and run_delphi
  without coordination (complete_run was already idempotent/crash-healing).
- scripts/job_poller.py — mirror_job_claimed / mirror_job_finished
  (module-level, testable without a JobProcessor): the OLD queue remains the
  single master (§6.2 invariant 3); v2 run rows are mirrors that become
  manifests as jobs execute. complete_job is the single funnel for every
  terminal path, so one hook covers success/failure/timeout/critical.
  Mirror errors are logged, NEVER raised — M1 requires the old serving path
  to survive a broken v2 store. Narrative job types deliberately not
  mirrored yet (P7d).
- run_delphi.py — write-mode-driven behavior: old = today's; both/v2 =
  manifest + input snapshot + fingerprints merged into the manifest +
  completion/failure marks at every exit path. Failure policy: both →
  log-and-continue (old path keeps serving; unreplayability is visible as
  missing fingerprints); v2 or explicit --snapshot-inputs → abort. Replays
  (--input-source) never write manifests here (the replay harness, P12,
  creates its own fresh run).
- delphi_storage/purge.py + scripts/delphi_purge.py — per-zid purge
  (list_runs + purge_job per run), completing the §9 GDPR path promised in
  P6a: snapshot/artifact partitions removed; manifests and latest pointers
  retained (they carry only config/counts/hashes — retention machinery
  removes them in P14).

Tests (TDD; tests/test_write_mode_manifest.py, RED first): 23 tests —
write-mode parsing/fail-loud/truth table, manifest helper idempotence and
latest-flip, poller mirror for all modes/job types incl. the never-raise
policy, run_delphi mode matrix (old/both/v2 × capture success/failure,
explicit-flag abort, unset-mode fail-loud), per-zid purge isolation.
Existing run_delphi-invoking suites updated to set the now-required mode.

Full suite: 469 passed, 58 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed.

Stack: P1 #2597 … P6b #2603, P7a this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jucor
jucor force-pushed the jc/delphi-storage-v2-p7a branch from 14c9f43 to c3c973f Compare July 10, 2026 11:40
@jucor
jucor force-pushed the jc/delphi-storage-v2-p6b branch from 704b041 to b00eeab Compare July 10, 2026 11:40
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1108 259 77%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 306 206 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/pca.py 135 18 87%
pca_kmeans_rep/repness.py 208 9 96%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 720 19%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 138 94 32%
run_math_pipeline.py 288 79 73%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 124 55 56%
umap_narrative/502_calculate_priorities.py 139 110 21%
umap_narrative/700_datamapplot_for_layer.py 503 469 7%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 802 740 8%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 0 100%
umap_narrative/llm_factory_constructor/model_provider.py 192 165 14%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 358 283 21%
umap_narrative/polismath_commentgraph/utils/storage.py 584 458 22%
umap_narrative/reset_conversation.py 165 50 70%
umap_narrative/run_pipeline.py 466 323 31%
utils/general.py 62 41 34%
Total 10957 7264 34%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jucor