feat(server): storage v2 P3 — TypeScript delphi storage twin (dual backends + jest conformance) by jucor · Pull Request #2599 · compdemocracy/polis

jucor · 2026-07-06T15:07:40Z

Stack 1 / P3 of the Storage V2 effort — the TypeScript twin of #2598, per delphi/docs/STORAGE_V2_DESIGN.md §3.6: the abstraction must be two-sided, because both the Python pipeline and the server touch the store directly.

Stacked on #2598 (jc/delphi-storage-v2-p2) — it needs the conformance case files from that PR at test time (same repo checkout, no build coupling).

What this adds

server/src/storage/delphi/ — the TypeScript implementation of the same backend-neutral repository, kept honest by executing the same JSON conformance case files (delphi/delphi_storage/conformance/cases/) as the pytest suite:

errors.ts, keys.ts, codec.ts — same wire format: canonical JSON (keys sorted by UTF-8 byte order, matching Python's sort_keys), little-endian packed float64, zstd via Node's built-in zlib zstd support (Node ≥ 22.15; no new npm dependency). The jest suite decodes zstd blobs that were encoded by Python and committed as fixtures — the cross-language pin.
interface.ts — method names camelCase, data fields snake_case (manifests/pointers are shared wire objects written by Python and read here, and vice versa); normalizeRunManifest mirrors the Pydantic validation; BaseDelphiStore carries the shared completeRun (scopes derived before the status flip; pointer written last as the commit point; idempotent + crash-healing) and appendLog semantics.
dynamoStore.ts — SDK v3 DynamoDBDocumentClient without convertEmptyValues (deliberate deviation from the older house pattern: empty strings must round-trip); optimistic-lock conditional writes on version; sparse claim-index GSI; generation-tagged chunk rows for blobs >300KB with the main item as the atomic commit point — the same crash-safe protocol as the Python backend (see the review fixes on feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance) #2598).
postgresStore.ts — pg Pool, COLLATE "C" sort keys, FOR UPDATE SKIP LOCKED claims, JSONB manifests; DDL identical to the Python backend's (migration 000019 in P4 materializes it).
memoryStore.ts — reference implementation (internally synchronous, so concurrent async callers can't interleave inside an op).
factory.ts + config.ts — DELPHI_STORAGE_BACKEND / DELPHI_STORAGE_TABLE_PREFIX / DELPHI_STORAGE_PG_SCHEMA / DELPHI_STORAGE_PG_URL, env access confined to config.ts per the lint rule.

Tests

__tests__/setup/delphi-storage-conformance.ts — the jest case runner (twin of runner.py).
Unit (no services): codec/keys pins (claim order, timestamps, scopes, canonical JSON, f64 hex, envelopes) + all 8 shared cases against the memory backend + Python-encoded codec fixtures. 37 tests.
Integration (services as in CI): all cases + a Promise.all two-claimants-one-wins race against real DynamoDB and PostgreSQL. 16 tests.

Testing

37 unit + 16 integration tests pass (DynamoDB local :8002, dockerized PG 16).
tsc build clean; eslint clean (0 errors).
The 12 pre-existing JWT-env unit-test failures reproduce identically on the base commit without this change (missing local JWT keys; CI provides them) — not a regression.
TDD: test suites written first (2 suites failing on missing modules), then the implementation.

Stack

P1: feat(delphi): Clojure-parity math fixes + storage-v2/replay design (s… #2597 (design doc)
P2: feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance) #2598 (Python delphi_storage/)
P3: this PR
P4: DDL — create_dynamodb_tables.py additions + PG migration 000019 (next)

🤖 Generated with Claude Code

… PG migration 000019) Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment artifacts for the V2 schema, pinned to the backend definitions. - create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables() wired into create_tables() (always created, like the job queue). The schemas are an INLINED copy because the script runs standalone in the dynamodb-init container (bare python + boto3, cannot import delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts from delphi_storage.backends.dynamodb.table_schemas("Delphi2_"). - server/postgres/migrations/000019_create_delphi_storage.sql: schema `delphi` with runs / latest / run_inputs / artifacts / topic_moderation / collective_statements, mirroring delphi_storage.backends.postgres.schema_ddl("delphi") statement for statement (drift-tested the same way). - tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the ensure_* test helpers) yield conformant stores: the migration applied by raw SQL to a scratch database passes the full conformance suite with ensure_schema off, and script-created Delphi2_* tables pass it with ensure_tables off. Service-gated tests follow the skip-locally / fail-in-GitHub-Actions convention. Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG all 6 DDL tests pass. Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1 (Foundation); Stack 2 (provenance plumbing, P5-P8) follows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jucor · 2026-07-06T15:23:10Z

Post-push self-review (Claude subagent) found one real divergence from the Python twin, now fixed and re-pushed across the stack:

Unvalidated status/job_type strings — updateRunStatus, advanceLatest, and listRuns accepted arbitrary strings and would silently persist them (Python raises via enum coercion). A typo'd status would have made a run permanently unclaimable with no error surfaced. Fixed symmetrically:

TS: coerceRunStatus / coerceJobType helpers in interface.ts, called at the top of all three methods in all three backends.
Python (feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance) #2598): the bare ValueError from RunStatus(status) / JobType(job_type) now maps to the contract's Invalid error via the same helpers.
Conformance: validation.json gained ops pinning this (update_run_status BOGUS → invalid with the run left untouched, advance_latest with a bogus job_type → invalid with no pointer written, list_runs with a bogus status / both zid+rid / neither → invalid) — exactly the case the reviewer noted would previously pass pytest but fail jest.

All suites re-verified: Python 43/43 storage tests across 4 backends; TS 53/53 (unit + integration); tsc + eslint clean.

… PG migration 000019) Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment artifacts for the V2 schema, pinned to the backend definitions. - create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables() wired into create_tables() (always created, like the job queue). The schemas are an INLINED copy because the script runs standalone in the dynamodb-init container (bare python + boto3, cannot import delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts from delphi_storage.backends.dynamodb.table_schemas("Delphi2_"). - server/postgres/migrations/000019_create_delphi_storage.sql: schema `delphi` with runs / latest / run_inputs / artifacts / topic_moderation / collective_statements, mirroring delphi_storage.backends.postgres.schema_ddl("delphi") statement for statement (drift-tested the same way). - tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the ensure_* test helpers) yield conformant stores: the migration applied by raw SQL to a scratch database passes the full conformance suite with ensure_schema off, and script-created Delphi2_* tables pass it with ensure_tables off. Service-gated tests follow the skip-locally / fail-in-GitHub-Actions convention. Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG all 6 DDL tests pass. Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1 (Foundation); Stack 2 (provenance plumbing, P5-P8) follows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… pipeline entry points Implements Stack 2 / P5 of docs/STORAGE_V2_DESIGN.md (§4.4): the pipeline job id travels ON THE COMMAND LINE instead of the env-var-only propagation that reached 2 of 18 tables (§3.2). - delphi_storage/job_id.py — resolve_job_id: explicit CLI value > DELPHI_JOB_ID env (transition fallback, removed in a later phase) > auto local-<uuid4> for dev runs. - run_delphi.py — gains --job-id, resolves once, exports DELPHI_JOB_ID for un-migrated env readers (same id as the command lines), and appends --job-id=<id> to all six stage subprocesses (reset, math, umap 500s, 501, 502, 700 per layer). - scripts/job_poller.py — command construction extracted into module-level build_job_command() (now unit-testable); FULL_PIPELINE and CREATE_NARRATIVE_BATCH commands carry --job-id=<queue job_id>; the DELPHI_JOB_ID env export stays for the transition window. 803's existing --job-id (the batch-tracking id, a distinct concept) is unchanged. - Stage entry points all accept --job-id: reset_conversation (CLI moved into cli(), main() stays the worker), run_math_pipeline (arg + log only — no math-logic change), run_pipeline (threaded into process_conversation, replacing the buried env read; auto prefix changes pipeline_run_<uuid> → local-<uuid> for bare dev runs), 501, 502 (arg + log, used from P7), 700 (threaded down to the S3-key builder; 'unknown' fallback preserved), 801 (arg > env, NO auto-generation — narrative section keys embed the id, so a missing one must keep failing loudly downstream). Tests (TDD; tests/test_job_id_threading.py, RED first): - resolve_job_id precedence; run_delphi threads one shared id to all six stage commands (explicit, auto, and env-fallback variants); - every stage entry point's --help advertises --job-id; - process_conversation exposes the job_id parameter; - poller commands for all three job types (FULL_PIPELINE + 801 carry the queue job id; 803 keeps batch_job_id semantics). Full suite: 424 passed, 30 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — math outputs untouched. Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… + purge tool Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2 entity 2, §4.4): reproducibility begins with recording EXACTLY what the pipeline read. The read seam (--input-source, P6b) follows. - delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid): writes the six snapshot kinds as run_inputs items (codec envelopes, zstd-compressed votes) and returns per-kind fingerprints (sha256 of the canonical uncompressed payload, row_count, max vote created — the P7 manifest inputs). Key fidelity decisions, pinned by tests: - votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs (the production math path never flips — fetch_votes()'s flip is dead code), superseded votes included, order preserved (KMeans init depends on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md). - The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings unchanged; golden suite + the e2e SQL pins stay green). - comments/participants captured in FULL, including moderated-out rows (mod = -1) and banned participants — filtering is pipeline behavior, not snapshot behavior. - clojure_math_main captured for ALL math_env rows (501 reads env-less latest, 801 filters MATH_ENV with a different default — snapshotting every env keeps both replayable). - derive_votes_latest_unique() reproduces PG's RULE-maintained votes_latest_unique from the stream (last vote per (pid, tid) in stream order) — tested against a seeded PG table mimicking the RULE. - delphi_storage/purge.py — purge_job(): deletes every generic-entity partition for a job (the §9 GDPR path for snapshot copies). - run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1): captures before any stage runs, aborts the run on capture failure (a run without recorded inputs defeats the point when enabled). Off by default until P7's DELPHI_WRITE_MODE subsumes it. Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw stream order/signs, compression, fingerprints, derivation-vs-PG-RULE, full-row comments/participants, all-env math_main, SQL-parity-by-import, purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests use a seeded scratch database (skip-locally / fail-in-CI convention). Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — math outputs untouched. Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Adds the TypeScript “twin” implementation of Delphi Storage V2 under server/src/storage/delphi/, matching the Python repository abstraction and validating cross-language behavior by executing the shared JSON conformance case files in Jest.

Changes:

Implemented Delphi Storage V2 repository interface + shared semantics (completeRun, appendLog) in TypeScript.
Added DynamoDB, PostgreSQL, and in-memory backends with queue/lease/latest-pointer semantics and KV entity support.
Added Jest unit + integration coverage that runs the shared conformance case scripts (and codec fixtures) against the TS backends.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
server/src/storage/delphi/postgresStore.ts	PostgreSQL backend implementation and schema DDL helpers.
server/src/storage/delphi/memoryStore.ts	In-memory reference backend used by unit conformance tests.
server/src/storage/delphi/keys.ts	Shared key/scope construction and UTF-8 byte-order compare utilities.
server/src/storage/delphi/interface.ts	Backend-neutral interface, manifest normalization/validation, and shared semantics (complete/log).
server/src/storage/delphi/factory.ts	Backend selection wiring via server config/env.
server/src/storage/delphi/errors.ts	Cross-language error classes and `code` mapping for conformance cases.
server/src/storage/delphi/dynamoStore.ts	DynamoDB backend with optimistic locking, claim-index querying, and chunked blob storage.
server/src/storage/delphi/codec.ts	Canonical JSON + f64 packing + zstd envelope codec (TS twin of Python).
server/src/config.ts	Adds Delphi storage env/config fields (backend selection + connection settings).
server/tests/unit/delphiStorage.memory.test.ts	Runs shared conformance cases + codec fixtures against memory backend.
server/tests/unit/delphiStorage.codec.test.ts	Unit tests for codec + key semantics.
server/tests/setup/delphi-storage-conformance.ts	Jest runner executing shared JSON operation-script case files.
server/tests/integration/delphiStorage.backends.test.ts	Conformance + race tests against real DynamoDB local and PostgreSQL.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+  async ensureSchema(): Promise<void> {
+    const client = await this.pool.connect();
+    try {
+      for (const statement of schemaDdl(this.schema)) {
+        await client.query(statement);
+      }
+    } finally {
+      client.release();
+    }
+  }


+    } catch (error) {
+      await client.query("ROLLBACK");
+      throw error;
+    } finally {
+      client.release();
+    }


…ckends + jest conformance) Implements Stack 1 / P3 of delphi/docs/STORAGE_V2_DESIGN.md: the TypeScript side of the two-sided storage abstraction (§3.6 — both the Python pipeline AND the server touch the store), kept honest by executing the SAME JSON conformance case files as the pytest suite (delphi/delphi_storage/conformance/cases/). - server/src/storage/delphi/: errors, keys, codec (canonical JSON with UTF-8-byte-order key sort, little-endian packed f64, zstd via Node's built-in zlib — Node >= 22.15), interface (snake_case wire fields shared with Python, normalizeRunManifest validation, BaseDelphiStore with the shared completeRun/appendLog semantics), memoryStore, dynamoStore, postgresStore, factory. - dynamoStore: SDK v3 DocumentClient WITHOUT convertEmptyValues (empty strings must round-trip), conditional writes on version, sparse claim-index GSI, generation-tagged chunk rows for blobs >300KB with the main item as commit point (crash-safe, matching the Python backend). - postgresStore: pg Pool, COLLATE "C" byte-order keys, FOR UPDATE SKIP LOCKED claims, JSONB manifests — same DDL as the Python backend and the upcoming migration 000019 (P4). - config.ts: delphiStorageBackend / delphiStorageTablePrefix / delphiStoragePgSchema / delphiStoragePgUrl (env read stays confined to config.ts per the lint rule). - __tests__/setup/delphi-storage-conformance.ts: the jest case runner (twin of runner.py). Unit: codec/keys pins + all 8 shared cases on the memory backend + Python-encoded zstd fixture decoding. Integration: all cases + a Promise.all two-claimants race against real DynamoDB and PostgreSQL. Tests: 37 unit + 16 integration pass (DynamoDB local :8002, dockerized PG 16); tsc build clean; eslint clean. The 12 pre-existing JWT-env unit failures reproduce identically on the base commit (missing local JWT keys, unrelated). Part of the storage-v2 stack: P1 #2597 (design), P2 #2598 (Python package), P4 (DDL) next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jucor mentioned this pull request Jul 6, 2026

feat(delphi): storage v2 P4 — deployable DDL (Delphi2 Dynamo tables + PG migration 000019) #2600

Open

jucor force-pushed the jc/delphi-storage-v2-p3 branch from 57be28e to 9ee292b Compare July 6, 2026 15:22

jucor force-pushed the jc/delphi-storage-v2-p2 branch from d87422d to 328f1de Compare July 6, 2026 15:22

jucor mentioned this pull request Jul 6, 2026

feat(delphi): storage v2 P5 — explicit --job-id threading through all pipeline entry points #2601

Open

jucor mentioned this pull request Jul 6, 2026

feat(delphi): storage v2 P6a — per-run input snapshots (capture side) + purge tool #2602

Open

jucor force-pushed the jc/delphi-storage-v2-p2 branch from 328f1de to fd5037e Compare July 6, 2026 18:48

jucor force-pushed the jc/delphi-storage-v2-p3 branch from 9ee292b to 5edc451 Compare July 6, 2026 18:48

jucor mentioned this pull request Jul 6, 2026

feat(delphi): storage v2 P6b — --input-source read seam + implementation notes #2603

Open

jucor requested a review from Copilot July 6, 2026 23:04

Copilot started reviewing on behalf of jucor July 6, 2026 23:04 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

jucor force-pushed the jc/delphi-storage-v2-p3 branch from 5edc451 to a8ff600 Compare July 6, 2026 23:58

jucor force-pushed the jc/delphi-storage-v2-p2 branch from fd5037e to 3c07e95 Compare July 6, 2026 23:58

This was referenced Jul 7, 2026

feat(delphi): storage v2 P7a — DELPHI_WRITE_MODE + run-manifest lifecycle + per-zid purge #2604

Open

feat(delphi): storage v2 P7b — math stage dual-write + verify_dual_write parity checker #2605

Open

jucor force-pushed the jc/delphi-storage-v2-p2 branch from 3c07e95 to afbe110 Compare July 10, 2026 11:40

jucor force-pushed the jc/delphi-storage-v2-p3 branch from a8ff600 to 9ff55da Compare July 10, 2026 11:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(server): storage v2 P3 — TypeScript delphi storage twin (dual backends + jest conformance)#2599

feat(server): storage v2 P3 — TypeScript delphi storage twin (dual backends + jest conformance)#2599
jucor wants to merge 1 commit into
jc/delphi-storage-v2-p2from
jc/delphi-storage-v2-p3

jucor commented Jul 6, 2026

Uh oh!

jucor commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jucor commented Jul 6, 2026

What this adds

Tests

Testing

Stack

Uh oh!

jucor commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants