Skip to content

feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance)#2598

Open
jucor wants to merge 1 commit into
edgefrom
jc/delphi-storage-v2-p2
Open

feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance)#2598
jucor wants to merge 1 commit into
edgefrom
jc/delphi-storage-v2-p2

Conversation

@jucor

@jucor jucor commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Stack 1 / P2 of the Storage V2 effort — design doc in #2597 (delphi/docs/STORAGE_V2_DESIGN.md, §4.2–4.3, §7).

What this adds

A new Python package delphi/delphi_storage/: the backend-neutral repository over the six V2 logical entities (runs, run_inputs, artifacts, latest, topic_moderation, collective_statements), with both production backends and the shared conformance spec that will keep the Python and TypeScript implementations honest.

Interface (interface.py)

  • Generic ops: put / put_batch / get / query_prefix / query_between / delete_partition — only for the four KV entities; runs/latest are reachable exclusively through semantic ops so their invariants can't be bypassed.
  • Semantic queue/run ops: enqueue_run, claim_next_run (priority desc → FIFO → job_id, conditional-write atomicity), extend_lease, update_run_status, merge_run_fields, complete_run, append_log, advance_latest, get_latest, list_runs.
  • complete_run writes the latest pointer last (the commit point, design §4.2), is idempotent, and heals a crash between manifest-COMPLETED and pointer flip.
  • advance_latest(only_if_absent_or_imported=True) implements the backfill importer's no-clobber invariant (§6.2 invariant 1) — conformance-tested now, used by P9.

Codec (codec.py)

Canonical JSON (sorted keys, compact, UTF-8, NaN rejected), little-endian packed float64 (bit-exact), zstd, and versioned payload envelopes (json / json+zstd / f64+zstd) carrying sha256 + byte counts of the uncompressed payload (compressed bytes are never compared: zstd output is implementation-dependent). Decimal↔float conversion for DynamoDB follows the existing Decimal(str(x)) convention.

Backends

  • DynamoDB (backends/dynamodb.py): thread-safe low-level client + TypeSerializer; optimistic-lock conditional writes on version (the lift of the working job_poller.py claim logic); sparse claim-index GSI for queue ordering; zid-index/rid-index for list_runs; blobs >300KB transparently chunked across hidden rows (sk prefixed U+007F) and reassembled on read — invisible to queries, stale chunks cleaned on overwrite.
  • PostgreSQL (backends/postgres.py): schema-scoped tables (default delphi), COLLATE "C" sort keys (= DynamoDB byte order), FOR UPDATE SKIP LOCKED claims, JSONB manifests following the legacy math_main pattern. The DDL in this file is the single Python-side source that migration 000019 (P4) mirrors.
  • Memory (backends/memory.py): in-process reference implementation / executable spec.
  • factory.py: DELPHI_STORAGE_BACKEND=dynamodb|postgres|memory (+ DELPHI_STORAGE_TABLE_PREFIX, DELPHI_STORAGE_PG_SCHEMA, DELPHI_STORAGE_PG_URL), reusing the existing DYNAMODB_ENDPOINT/AWS_REGION/DATABASE_URL vocabulary.

Conformance suite (conformance/)

  • Normative spec in conformance/README.md + 8 JSON operation-script case files — the same files P3's jest suite executes against the TS backends.
  • Coverage: unicode/float/null round-trips, UTF-8 byte-order prefix/between queries (incl. an astral-plane case that catches UTF-16 comparison bugs), chunk-spanning blob round-trips + overwrite-shrink cleanup, queue claim ordering/leases/duplicate-enqueue, monotonic latest seq, idempotent completion, importer no-clobber, validation rejections, and cross-language codec fixtures (Python-encoded zstd blobs the TS side must decode).
  • pytest runs every case against all four backend params: memory and moto always; real dynamodb/postgres skip locally when unreachable and fail loudly under GITHUB_ACTIONS (both services are up in CI).
  • A threaded two-claimants-one-wins race test runs against memory + both real backends.

Testing

  • Full suite: 398 passed, 28 skipped, 58 xfailed (baseline before this PR: 332/12/58; the 5 pre-existing DynamoDB-unavailable errors unchanged).
  • With DynamoDB local (:8002) + dockerized PG 16: all backend×case params green, including the race tests.
  • TDD: cases + tests written first (RED: 3 collection errors), then the package (GREEN).

Dependencies

Adds zstandard>=0.23 to pyproject.toml and the matching pin to requirements.lock (Docker build).

Stack

🤖 Generated with Claude Code

@jucor jucor force-pushed the jc/delphi-storage-v2-p2 branch from 2ec6794 to d87422d Compare July 6, 2026 15:02
@jucor

jucor commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Post-push self-review (Claude subagent) found two real issues, now fixed and re-pushed:

  1. Chunked-blob overwrite was not crash-safe (DynamoDB backend) — the old write order (delete old chunks → write new chunks → flip main item) could permanently corrupt the previous value on a mid-write crash, and concurrent readers could see spurious corruption errors. Now: chunk rows are generation-tagged; a put writes the new generation's rows first (invisible), then flips the main item (the commit point), then sweeps stale generations. Reads fetch exactly the committed generation's deterministic chunk keys. New moto tests in tests/test_delphi_storage_dynamodb.py cover the crashed-write window, stale-generation injection, shrink-overwrite, and same-size overwrite.

  2. complete_run could strand a malformed run — it flipped status to COMPLETED before deriving latest scopes, so a FULL_PIPELINE run without zid ended up completed-but-never-published, with a raw ValueError escaping the StorageError contract. Now the job_type↔zid/rid coupling is validated at enqueue_run (invalid), and complete_run derives scopes before the status flip. Conformance cases extended accordingly (plus scalar-field merge_run_fields coverage).

Full suite: 402 passed, 28 skipped, 58 xfailed (same 5 pre-existing DynamoDB-unavailable errors as baseline); all four backend conformance params green against DynamoDB local + dockerized PG.

jucor added a commit that referenced this pull request Jul 6, 2026
… PG migration 000019)

Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment
artifacts for the V2 schema, pinned to the backend definitions.

- create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables()
  wired into create_tables() (always created, like the job queue). The
  schemas are an INLINED copy because the script runs standalone in the
  dynamodb-init container (bare python + boto3, cannot import
  delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts
  from delphi_storage.backends.dynamodb.table_schemas("Delphi2_").
- server/postgres/migrations/000019_create_delphi_storage.sql: schema
  `delphi` with runs / latest / run_inputs / artifacts / topic_moderation /
  collective_statements, mirroring
  delphi_storage.backends.postgres.schema_ddl("delphi") statement for
  statement (drift-tested the same way).
- tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the
  ensure_* test helpers) yield conformant stores: the migration applied by
  raw SQL to a scratch database passes the full conformance suite with
  ensure_schema off, and script-created Delphi2_* tables pass it with
  ensure_tables off. Service-gated tests follow the skip-locally /
  fail-in-GitHub-Actions convention.

Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG
all 6 DDL tests pass.

Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1
(Foundation); Stack 2 (provenance plumbing, P5-P8) follows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jucor jucor force-pushed the jc/delphi-storage-v2-p2 branch from d87422d to 328f1de Compare July 6, 2026 15:22
jucor added a commit that referenced this pull request Jul 6, 2026
… PG migration 000019)

Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment
artifacts for the V2 schema, pinned to the backend definitions.

- create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables()
  wired into create_tables() (always created, like the job queue). The
  schemas are an INLINED copy because the script runs standalone in the
  dynamodb-init container (bare python + boto3, cannot import
  delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts
  from delphi_storage.backends.dynamodb.table_schemas("Delphi2_").
- server/postgres/migrations/000019_create_delphi_storage.sql: schema
  `delphi` with runs / latest / run_inputs / artifacts / topic_moderation /
  collective_statements, mirroring
  delphi_storage.backends.postgres.schema_ddl("delphi") statement for
  statement (drift-tested the same way).
- tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the
  ensure_* test helpers) yield conformant stores: the migration applied by
  raw SQL to a scratch database passes the full conformance suite with
  ensure_schema off, and script-created Delphi2_* tables pass it with
  ensure_tables off. Service-gated tests follow the skip-locally /
  fail-in-GitHub-Actions convention.

Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG
all 6 DDL tests pass.

Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1
(Foundation); Stack 2 (provenance plumbing, P5-P8) follows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor added a commit that referenced this pull request Jul 6, 2026
… pipeline entry points

Implements Stack 2 / P5 of docs/STORAGE_V2_DESIGN.md (§4.4): the pipeline
job id travels ON THE COMMAND LINE instead of the env-var-only propagation
that reached 2 of 18 tables (§3.2).

- delphi_storage/job_id.py — resolve_job_id: explicit CLI value >
  DELPHI_JOB_ID env (transition fallback, removed in a later phase) >
  auto local-<uuid4> for dev runs.
- run_delphi.py — gains --job-id, resolves once, exports DELPHI_JOB_ID for
  un-migrated env readers (same id as the command lines), and appends
  --job-id=<id> to all six stage subprocesses (reset, math, umap 500s,
  501, 502, 700 per layer).
- scripts/job_poller.py — command construction extracted into module-level
  build_job_command() (now unit-testable); FULL_PIPELINE and
  CREATE_NARRATIVE_BATCH commands carry --job-id=<queue job_id>; the
  DELPHI_JOB_ID env export stays for the transition window. 803's existing
  --job-id (the batch-tracking id, a distinct concept) is unchanged.
- Stage entry points all accept --job-id: reset_conversation (CLI moved
  into cli(), main() stays the worker), run_math_pipeline (arg + log only —
  no math-logic change), run_pipeline (threaded into process_conversation,
  replacing the buried env read; auto prefix changes pipeline_run_<uuid> →
  local-<uuid> for bare dev runs), 501, 502 (arg + log, used from P7),
  700 (threaded down to the S3-key builder; 'unknown' fallback preserved),
  801 (arg > env, NO auto-generation — narrative section keys embed the id,
  so a missing one must keep failing loudly downstream).

Tests (TDD; tests/test_job_id_threading.py, RED first):
- resolve_job_id precedence; run_delphi threads one shared id to all six
  stage commands (explicit, auto, and env-fallback variants);
- every stage entry point's --help advertises --job-id;
- process_conversation exposes the job_id parameter;
- poller commands for all three job types (FULL_PIPELINE + 801 carry the
  queue job id; 803 keeps batch_job_id semantics).

Full suite: 424 passed, 30 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.

Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor added a commit that referenced this pull request Jul 6, 2026
… + purge tool

Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.

- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
  writes the six snapshot kinds as run_inputs items (codec envelopes,
  zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
  canonical uncompressed payload, row_count, max vote created — the P7
  manifest inputs). Key fidelity decisions, pinned by tests:
  - votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
    (the production math path never flips — fetch_votes()'s flip is dead
    code), superseded votes included, order preserved (KMeans init depends
    on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
  - The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
    VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
    unchanged; golden suite + the e2e SQL pins stay green).
  - comments/participants captured in FULL, including moderated-out rows
    (mod = -1) and banned participants — filtering is pipeline behavior,
    not snapshot behavior.
  - clojure_math_main captured for ALL math_env rows (501 reads env-less
    latest, 801 filters MATH_ENV with a different default — snapshotting
    every env keeps both replayable).
  - derive_votes_latest_unique() reproduces PG's RULE-maintained
    votes_latest_unique from the stream (last vote per (pid, tid) in stream
    order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
  partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
  captures before any stage runs, aborts the run on capture failure
  (a run without recorded inputs defeats the point when enabled).
  Off by default until P7's DELPHI_WRITE_MODE subsumes it.

Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).

Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.

Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jucor jucor force-pushed the jc/delphi-storage-v2-p2 branch from 328f1de to fd5037e Compare July 6, 2026 18:48
jucor added a commit that referenced this pull request Jul 6, 2026
… + purge tool

Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.

- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
  writes the six snapshot kinds as run_inputs items (codec envelopes,
  zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
  canonical uncompressed payload, row_count, max vote created — the P7
  manifest inputs). Key fidelity decisions, pinned by tests:
  - votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
    (the production math path never flips — fetch_votes()'s flip is dead
    code), superseded votes included, order preserved (KMeans init depends
    on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
  - The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
    VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
    unchanged; golden suite + the e2e SQL pins stay green).
  - comments/participants captured in FULL, including moderated-out rows
    (mod = -1) and banned participants — filtering is pipeline behavior,
    not snapshot behavior.
  - clojure_math_main captured for ALL math_env rows (501 reads env-less
    latest, 801 filters MATH_ENV with a different default — snapshotting
    every env keeps both replayable).
  - derive_votes_latest_unique() reproduces PG's RULE-maintained
    votes_latest_unique from the stream (last vote per (pid, tid) in stream
    order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
  partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
  captures before any stage runs, aborts the run on capture failure
  (a run without recorded inputs defeats the point when enabled).
  Off by default until P7's DELPHI_WRITE_MODE subsumes it.

Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).

Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.

Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jucor added a commit that referenced this pull request Jul 6, 2026
… + purge tool

Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.

- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
  writes the six snapshot kinds as run_inputs items (codec envelopes,
  zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
  canonical uncompressed payload, row_count, max vote created — the P7
  manifest inputs). Key fidelity decisions, pinned by tests:
  - votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
    (the production math path never flips — fetch_votes()'s flip is dead
    code), superseded votes included, order preserved (KMeans init depends
    on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
  - The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
    VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
    unchanged; golden suite + the e2e SQL pins stay green).
  - comments/participants captured in FULL, including moderated-out rows
    (mod = -1) and banned participants — filtering is pipeline behavior,
    not snapshot behavior.
  - clojure_math_main captured for ALL math_env rows (501 reads env-less
    latest, 801 filters MATH_ENV with a different default — snapshotting
    every env keeps both replayable).
  - derive_votes_latest_unique() reproduces PG's RULE-maintained
    votes_latest_unique from the stream (last vote per (pid, tid) in stream
    order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
  partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
  captures before any stage runs, aborts the run on capture failure
  (a run without recorded inputs defeats the point when enabled).
  Off by default until P7's DELPHI_WRITE_MODE subsumes it.

Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).

Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.

Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jucor jucor requested a review from Copilot July 6, 2026 23:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces Delphi Storage V2 as a backend-neutral Python repository (delphi_storage) with DynamoDB, PostgreSQL, and in-memory backends, plus a shared JSON conformance suite (operation-script case files) intended to match the upcoming TypeScript implementation.

Changes:

  • Adds delphi/delphi_storage/ implementing the V2 interface, codec, keying rules, and three backends (memory/DynamoDB/Postgres).
  • Adds a pytest conformance runner that executes shared JSON case files against all backends (plus backend-specific DynamoDB chunking tests).
  • Updates packaging/build artifacts (Hatch wheel packages, Docker copy step) and adds the zstandard dependency + lock pin.

Reviewed changes

Copilot reviewed 25 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
delphi/tests/test_delphi_storage_keys.py Unit tests for key builders (claim ordering, timestamp format, scopes, artifact keys).
delphi/tests/test_delphi_storage_dynamodb.py DynamoDB-specific tests for crash-safe chunked blob write behavior (moto).
delphi/tests/test_delphi_storage_conformance.py Pytest harness to run shared JSON operation-script conformance cases across backends (incl. race test).
delphi/tests/test_delphi_storage_codec.py Unit tests for canonical JSON, f64 packing, zstd compression, and payload envelopes.
delphi/requirements.lock Pins zstandard in the locked dependency set.
delphi/pyproject.toml Adds zstandard dependency and includes delphi_storage in wheel/sdist packaging.
delphi/Dockerfile Copies delphi_storage/ into builder and runtime images.
delphi/delphi_storage/init.py Exposes the public get_store entrypoint and core models/errors.
delphi/delphi_storage/models.py Defines Pydantic models for manifests, pointers, and KV items.
delphi/delphi_storage/keys.py Defines canonical timestamp validation and key/scope builders (cross-language contract).
delphi/delphi_storage/interface.py Defines the backend-neutral repository interface and shared contract validation helpers.
delphi/delphi_storage/factory.py Implements backend selection via env vars (`dynamodb
delphi/delphi_storage/codec.py Implements canonical JSON + zstd + packed-f64 payload envelopes and DynamoDB number helpers.
delphi/delphi_storage/backends/init.py Backend package marker.
delphi/delphi_storage/backends/memory.py In-memory reference backend used as an executable spec + test double.
delphi/delphi_storage/backends/dynamodb.py DynamoDB backend including chunked blob storage, optimistic locking, GSIs, and latest pointers.
delphi/delphi_storage/backends/postgres.py PostgreSQL backend with schema-scoped tables, COLLATE "C" ordering, and SKIP LOCKED claims.
delphi/delphi_storage/conformance/init.py Conformance package marker.
delphi/delphi_storage/conformance/README.md Normative conformance spec describing entities, operations, ordering, and case format.
delphi/delphi_storage/conformance/runner.py JSON case loader/executor and assertion utilities for conformance execution.
delphi/delphi_storage/conformance/cases/roundtrip_basic.json Case: basic put/get round-trips, unicode, numbers-by-value, overwrite.
delphi/delphi_storage/conformance/cases/query_ordering.json Case: UTF-8 byte-order prefix/between queries + delete_partition behavior.
delphi/delphi_storage/conformance/cases/blob_chunking.json Case: large blob round-trips + chunk invisibility + overwrite shrink cleanup.
delphi/delphi_storage/conformance/cases/queue_claim.json Case: claim ordering, leases, duplicate enqueue, terminal transitions, list_runs.
delphi/delphi_storage/conformance/cases/latest_pointer.json Case: monotonic latest seq, idempotent completion, importer no-clobber invariant, rid scopes.
delphi/delphi_storage/conformance/cases/run_manifest_fields.json Case: merge_run_fields provenance accumulation + append_log sequencing as artifacts.
delphi/delphi_storage/conformance/cases/validation.json Case: invalid writes rejected + runs/latest unreachable via generic ops + status/type validation.
delphi/delphi_storage/conformance/cases/codec_fixtures.json Case: cross-language codec fixtures and round-trips for envelope encodings.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +118 to +120
if enc == "json+zstd":
return json.loads(raw.decode("utf-8"))
return unpack_f64(raw)
Comment on lines +60 to +61
def _numbers_equal(a: Any, b: Any) -> bool:
return not isinstance(a, bool) and not isinstance(b, bool) and float(a) == float(b)
Comment on lines +8 to +30
import uuid

import pytest

moto = pytest.importorskip("moto")

from delphi_storage.backends.dynamodb import ( # noqa: E402 - after importorskip
CHUNK_SIZE,
DynamoDelphiStore,
_chunk_prefix,
_chunk_sk,
)
from delphi_storage.models import StoreItem # noqa: E402


@pytest.fixture()
def store():
with moto.mock_aws():
yield DynamoDelphiStore(
table_prefix=f"ChunkT{uuid.uuid4().hex[:8]}_",
region="us-east-1",
ensure_tables=True,
)
Comment on lines +651 to +655
rows = self._query_all(
TableName=self._runs_table,
IndexName=index,
KeyConditionExpression=f"{attr} = :key",
ExpressionAttributeValues={":key": {"S": value}},
Implements Stack 1 / P2 of docs/STORAGE_V2_DESIGN.md (§4.2-4.3): the neutral
repository interface over the six logical entities (runs, run_inputs,
artifacts, latest, topic_moderation, collective_statements), with BOTH
production backends and the shared conformance spec that keeps the Python and
TypeScript implementations honest.

- delphi_storage/interface.py — DelphiStore ABC: generic put/get/
  query_prefix/query_between/delete_partition + semantic queue ops
  (enqueue_run, claim_next_run, extend_lease, update_run_status,
  merge_run_fields, complete_run, append_log, advance_latest, get_latest,
  list_runs). complete_run flips the latest pointer last (commit point),
  idempotent and crash-healing; advance_latest carries the importer
  no-clobber mode (§6.2 invariant 1).
- delphi_storage/codec.py — canonical JSON, little-endian packed float64,
  zstd, payload envelopes (json / json+zstd / f64+zstd; sha256 + byte counts
  of the uncompressed payload), Decimal conversion for DynamoDB.
- delphi_storage/keys.py — claim-order strings (priority desc then FIFO),
  canonical millisecond-UTC timestamps, latest scopes, log keys.
- backends/dynamodb.py — thread-safe low-level client, conditional writes on
  version (lift of the job_poller optimistic lock), sparse claim GSI,
  transparent >300KB blob chunking behind the interface.
- backends/postgres.py — schema-scoped tables, COLLATE "C" byte-order keys,
  FOR UPDATE SKIP LOCKED claims; DDL is the single Python-side source that
  migration 000019 (P4) will mirror.
- backends/memory.py — in-process reference implementation.
- conformance/ — normative README + 8 shared JSON case files executed by
  pytest across ALL backends (memory always; moto always; real DynamoDB and
  PostgreSQL under the skip-locally/fail-in-CI convention), covering
  round-trips (unicode/floats/blobs), UTF-8 byte-order queries, chunk-spanning
  payloads, queue claim ordering + leases, monotonic latest pointers,
  idempotent completion, importer no-clobber, and validation. A threaded
  two-claimants-one-wins race test runs against memory + both real backends.
- Dockerfile — add delphi_storage/ to the builder and final COPY layers,
  the same way the sibling first-party packages polismath/ and umap_narrative/
  are baked into the image. Without it the image builds fine but
  `import delphi_storage` raises ModuleNotFoundError at pytest collection in
  the CI test stage (the package is declared in pyproject but its source was
  never copied into the image).
- backends/postgres.py — normalize the legacy `postgres://` URL scheme to
  `postgresql://` in PostgresDelphiStore. SQLAlchemy 2.0 dropped the
  `postgres://` alias and raises NoSuchModuleError on it, but that's still what
  many platforms (Heroku et al.) put in DATABASE_URL — so a production
  DATABASE_URL would otherwise crash engine creation. Only the scheme is
  rewritten; any +driver suffix is preserved. Unit-tested without a live DB
  (tests/test_delphi_storage_postgres.py; create_engine resolves the dialect
  eagerly but opens no connection).

Tests: 398 passed, 28 skipped, 58 xfailed (baseline: 332/12/58; the 5
pre-existing DynamoDB-unavailable errors unchanged). With DynamoDB local
(:8002) + dockerized PG: all 4 backends fully green.

Part of the storage-v2 stack (P1 = #2597, the design doc). P3 (TypeScript
twin) and P4 (DDL) follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jucor jucor force-pushed the jc/delphi-storage-v2-p2 branch from 3c07e95 to afbe110 Compare July 10, 2026 11:40
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1108 259 77%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 306 206 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/pca.py 135 18 87%
pca_kmeans_rep/repness.py 208 9 96%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 720 19%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 138 94 32%
run_math_pipeline.py 261 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 192 192 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 518 11%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/general.py 62 41 34%
Total 10873 7531 31%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants