feat(delphi): storage v2 P7c — umap 500s stage dual-write + verifier extension#2606
Open
jucor wants to merge 1 commit into
Open
feat(delphi): storage v2 P7c — umap 500s stage dual-write + verifier extension#2606jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
Delphi Coverage Report
|
…extension Implements the UMAP stage-group of Stack 2 / P7 (docs/STORAGE_V2_DESIGN.md §4.2, §6.1 M1, §7): the seven 500s outputs become v2 artifacts alongside the legacy Delphi_* tables, gated by DELPHI_WRITE_MODE. - umap_narrative/v2_artifacts.py — the dual-write seam is a DUCK-TYPED WRAPPER (DualWriteUmapStorage) over the legacy DynamoDBStorage: run_pipeline keeps calling the same seven write methods, and the wrapper feeds the SAME already-materialized Pydantic model lists to both stores. Serialize-once holds by construction — the models carry baked datetime fields (ConversationMeta.processed_date, LLMTopicName.created_at); re-instantiating them for a second path would be the math_tick trap again. Artifact keys per design §4.2: umap#meta, umap#embeddings#<chunk> (500/ chunk — fat vector rows), umap#graph#<chunk>, umap#assignments#<chunk> (5000/chunk), umap#keywords#<layer>, umap#features#<layer>, umap#topic#<layer>#<cluster>. Failure policy: both → logged never raised; v2 → raised. finalize() records stage_status.umap + a log line on the manifest (best-effort for standalone runs). - make_umap_storage — old mode returns the PLAIN legacy storage (zero overhead, byte-identical); both/v2 return the wrapper (v2-only with no legacy inner). - run_pipeline.py — write mode resolved up front (fail-fast, before process_comments and all writes); the storage-init block constructs the legacy client only when old writes are enabled and routes everything through the factory; ALL SEVEN call sites are untouched. finalize() after layer processing. - scripts/verify_dual_write.py — verify_umap_dual_write (independent reads of the seven conversation_id-keyed legacy tables vs decoded umap# artifacts; SAME-RUN comparisons only — EVōC cluster ids are not stable across runs); CLI gains --stage math|umap|all. Tests (TDD; tests/test_umap_dual_write.py, RED first): 12 tests — builders (chunking, per-layer grouping, per-cluster topic keys) from real Pydantic models, wrapper semantics (same-list dual writes, v2-only synthesis, failure policies, manifest finalize), factory matrix, run_pipeline wiring pin, and an end-to-end parity test against real DynamoDB local (legacy write via the REAL DynamoDBStorage, verify passes; corrupt umap#meta, verify catches it). Full suite: 493 passed, 60 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — write-side only. Stack: P1 #2597 … P7b #2605, P7c this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor
force-pushed
the
jc/delphi-storage-v2-p7c
branch
from
July 10, 2026 11:40
9073046 to
6e29d4e
Compare
jucor
force-pushed
the
jc/delphi-storage-v2-p7b
branch
from
July 10, 2026 11:40
dcab2b1 to
381d2cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 / P7c of the Storage V2 effort (design in #2597, §4.2 + §6.1 M1 + §7): the UMAP 500s stage-group dual-write — all seven stage outputs become v2 artifacts alongside the legacy tables, gated by
DELPHI_WRITE_MODE, with the verifier extended to cover them.Stacked on #2605 (
jc/delphi-storage-v2-p7b).The seam: a duck-typed wrapper, not seven edits
Unlike math (one funnel), the 500s stage writes through seven
DynamoDBStoragemethod calls. The seam isDualWriteUmapStorage(umap_narrative/v2_artifacts.py): it duck-types exactly those seven methods, so all seven call sites inrun_pipeline.pyare untouched — the wrapper feeds the same already-materialized Pydantic model lists to the legacy tables and the v2 artifact store.That makes the P7b serialize-once lesson hold by construction: the models carry baked
datetime.now()fields (ConversationMeta.processed_date,LLMTopicName.created_at); re-instantiating them for a second write path would tag the two stores differently — the math_tick trap all over again.Artifact keys per design §4.2:
umap#meta,umap#embeddings#<chunk>(500/chunk — the fat vector rows),umap#graph#<chunk>,umap#assignments#<chunk>(5000/chunk),umap#keywords#<layer>,umap#features#<layer>,umap#topic#<layer>#<cluster>(the job-id-versioned LLM names; note the legacy model silently drops itsjob_idkwarg — onlytopic_keycarries it, mirrored as-is).make_umap_storage:oldmode returns the plain legacy storage (zero overhead, byte-identical);both/v2return the wrapper (v2-only runs with no legacy inner).both→ v2 failures logged, never raised;v2→ raised.finalize()recordsstage_status.umap+ a log line on the run manifest (best-effort for manifest-less standalone runs).run_pipeline.py: write mode resolved up front (fail-fast — beforeprocess_commentsand all writes, per the P7b lesson); the legacy client is constructed only when old writes are enabled.Verifier extension
verify_umap_dual_writeinscripts/verify_dual_write.py— independent reads of the sevenconversation_id-keyed legacy tables vs the decodedumap#artifacts, numeric-tolerant. Same-run comparisons only: EVōC cluster ids are not stable across runs (unseeded until P8), but both writers consume the same in-memory arrays within a run, so parity holds by construction. CLI gains--stage math|umap|all.Testing (TDD — 12 tests written first, RED on missing module)
Builders (chunking, per-layer grouping, per-cluster topic keys) from real Pydantic models; wrapper semantics (same-list dual writes, v2-only synthesis of the legacy result shape, both failure policies, manifest finalize); factory matrix; a source-level wiring pin on
process_conversation; and an end-to-end parity test against real DynamoDB local — legacy write via the realDynamoDBStorage(tables created by the realcreate_evoc_tables), verify passes; corruptumap#meta, verify catches it.Full suite: 493 passed, 60 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged). Golden snapshots ran and passed — write-side only.
Stack
narrative#<section>#<model>artifacts, manifest mirroring for the narrative job types, numeric-rid resolution), then P7e 501/502/700 (prioritiesartifact kills the in-place mutation)🤖 Generated with Claude Code