feat(delphi): storage v2 P7b — math stage dual-write + verify_dual_write parity checker#2605
Open
jucor wants to merge 1 commit into
Open
feat(delphi): storage v2 P7b — math stage dual-write + verify_dual_write parity checker#2605jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
jucor
force-pushed
the
jc/delphi-storage-v2-p7b
branch
from
d53bbaa to
50f30e3
Compare
Collaborator
Author
|
Post-push self-review (Claude subagent) — findings triaged, fixes pushed:
Suite with services: 530 passed / 58 xfailed (only the 4 known legacy-table |
jucor
force-pushed
the
jc/delphi-storage-v2-p7b
branch
2 times, most recently
from
d950db4 to
dcab2b1
Compare
Delphi Coverage Report
|
…ite parity checker Implements the math stage-group of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md §4.2 artifacts, §6.1 M1, §7): the math results become v2 artifacts alongside the legacy Delphi_* tables, gated by DELPHI_WRITE_MODE. - polismath/database/v2_artifacts.py — build_math_artifacts derives the v2 payloads from the SAME objects the legacy writer consumes (conv.to_dynamo_dict() output, conv.proj, conv.group_clusters), converting Decimals back to floats by value. Artifact keys per design §4.2: math#pca (incl. the FULL consensus dict — legacy keeps only the agree list), math#kmeans, math#repness, math#routing#<chunk>, math#projections#<chunk> (logical chunks of 5000 participants; 18k+ happen in production — layered over the backend's byte-level auto-chunking). LEGACY QUIRKS REPRODUCED DELIBERATELY, each pinned by a test: routing priority/consensus default to 0 (never omitted), and the participant→group map is RAW-keyed exactly like the legacy writer — a str/int key mismatch yields group_id -1 on BOTH sides. - write_math_artifacts stores codec envelopes under pk=job_id, records math_tick_legacy on the manifest and appends a log line (best-effort for standalone runs without a manifest row). - run_math_pipeline.py — the write block becomes persist_math_results(conv, job_id, write_mode): old → legacy export only (unchanged behavior, extracted verbatim into _export_to_legacy_dynamo); both → legacy AND v2, v2 failures logged never raised (old path keeps serving); v2 → v2 only, failures exit 1 so the poller marks the run FAILED. - scripts/verify_dual_write.py — the M1 parity gate: reads the legacy tables (at latest_math_tick) and the decoded v2 artifacts back INDEPENDENTLY (zero shared code with the writer) and compares numerically (Decimal vs float by value). CLI: --job-id/--zid/--endpoint-url, exit 0/1. During development it caught three real divergences in the first builder draft (consensus agree-list only, routing defaults, raw-keyed group map) — exactly the failure mode it exists to catch. Tests (TDD; tests/test_math_dual_write.py, RED first): 12 tests — builder payloads/joins/chunking/quirks, envelope round-trip + manifest bookkeeping, mode gating incl. failure policies, and an end-to-end parity test against real DynamoDB local (legacy write via the REAL DynamoDBClient, then verify passes; corrupt one v2 value, verify catches it). Skip-locally / fail-in-CI convention. Full suite: 480 passed, 59 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — math outputs untouched (write-side only). Stack: P1 #2597 … P7a #2604, P7b this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
added a commit
that referenced
this pull request
Jul 10, 2026
…extension Implements the UMAP stage-group of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md §4.2, §6.1 M1, §7): the seven 500s outputs become v2 artifacts alongside the legacy Delphi_* tables, gated by DELPHI_WRITE_MODE. - umap_narrative/v2_artifacts.py — the dual-write seam is a DUCK-TYPED WRAPPER (DualWriteUmapStorage) over the legacy DynamoDBStorage: run_pipeline keeps calling the same seven write methods, and the wrapper feeds the SAME already-materialized Pydantic model lists to both stores. Serialize-once holds by construction — the models carry baked datetime fields (ConversationMeta.processed_date, LLMTopicName.created_at); re-instantiating them for a second path would be the math_tick trap again. Artifact keys per design §4.2: umap#meta, umap#embeddings#<chunk> (500/ chunk — fat vector rows), umap#graph#<chunk>, umap#assignments#<chunk> (5000/chunk), umap#keywords#<layer>, umap#features#<layer>, umap#topic#<layer>#<cluster>. Failure policy: both → logged never raised; v2 → raised. finalize() records stage_status.umap + a log line on the manifest (best-effort for standalone runs). - make_umap_storage — old mode returns the PLAIN legacy storage (zero overhead, byte-identical); both/v2 return the wrapper (v2-only with no legacy inner). - run_pipeline.py — write mode resolved up front (fail-fast, before process_comments and all writes); the storage-init block constructs the legacy client only when old writes are enabled and routes everything through the factory; ALL SEVEN call sites are untouched. finalize() after layer processing. - scripts/verify_dual_write.py — verify_umap_dual_write (independent reads of the seven conversation_id-keyed legacy tables vs decoded umap# artifacts; SAME-RUN comparisons only — EVōC cluster ids are not stable across runs); CLI gains --stage math|umap|all. Tests (TDD; tests/test_umap_dual_write.py, RED first): 12 tests — builders (chunking, per-layer grouping, per-cluster topic keys) from real Pydantic models, wrapper semantics (same-list dual writes, v2-only synthesis, failure policies, manifest finalize), factory matrix, run_pipeline wiring pin, and an end-to-end parity test against real DynamoDB local (legacy write via the REAL DynamoDBStorage, verify passes; corrupt umap#meta, verify catches it). Full suite: 493 passed, 60 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — write-side only. Stack: P1 #2597 … P7b #2605, P7c this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p7a
branch
from
14c9f43 to
c3c973f
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p7b
branch
from
dcab2b1 to
381d2cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 / P7b of the Storage V2 effort (design in #2597, §4.2 artifacts + §6.1 M1 + §7): the math stage-group dual-write — math results become v2 artifacts alongside the legacy
Delphi_*tables, gated byDELPHI_WRITE_MODE, with the design'sverify_dual_writeparity gate.Stacked on #2604 (
jc/delphi-storage-v2-p7a).What this adds
Artifact builder (
polismath/database/v2_artifacts.py)Derives the v2 payloads from the same objects the legacy writer consumes (
conv.to_dynamo_dict()output,conv.proj,conv.group_clusters), converting Decimals back to floats by value. Keys per design §4.2:math#pca(carrying the full consensus dict — legacy keeps only the agree list),math#kmeans,math#repness,math#routing#<chunk>,math#projections#<chunk>(logical chunks of 5000 participants — 18k+ happen in production — layered on the backend's byte-level auto-chunking).Legacy quirks reproduced deliberately, each pinned by a test:
priority/consensus_scoredefault to0(the legacy rows always carry both attributes);group_id: -1on both sides rather than being "fixed" on one.The persistence seam (
run_math_pipeline.py::persist_math_results)old→ legacy export only (the previous write block extracted verbatim into_export_to_legacy_dynamo; behavior unchanged, including its historical log-and-continue).both→ legacy AND v2; v2 failures are logged, never raised (M1: the old path keeps serving).v2→ v2 only; failures exit 1 so the poller marks the run FAILED.The parity gate (
scripts/verify_dual_write.py)Reads the legacy tables (at
latest_math_tick) and the decoded v2 artifacts back independently — zero shared code with the writer — and compares numerically (Decimal vs float by value). CLI:--job-id/--zid/--endpoint-url, exit 0/1.It proved itself during development: the first builder draft failed verification with three real divergences (the consensus agree-list subset, the routing defaults, the raw-keyed group map) — exactly the class of bug it exists to catch, found before any reviewer saw the code.
Testing (TDD — 12 tests written first, RED on missing modules)
Builder payloads/joins/chunking/quirk-parity; envelope round-trip +
math_tick_legacymanifest bookkeeping (tolerant of manifest-less standalone runs); mode gating incl. both failure policies; and an end-to-end parity test against real DynamoDB local — legacy write via the realDynamoDBClient, verify passes; corrupt one v2 value, verify catches it (skip-locally / fail-in-CI convention).Full suite: 480 passed, 59 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged). Golden snapshots ran and passed — write-side only, math outputs untouched.
Stack
🤖 Generated with Claude Code