feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance)#2598
feat(delphi): storage v2 P2 — backend-neutral delphi_storage repository (dual backends + conformance)#2598jucor wants to merge 1 commit into
Conversation
2ec6794 to
d87422d
Compare
|
Post-push self-review (Claude subagent) found two real issues, now fixed and re-pushed:
Full suite: 402 passed, 28 skipped, 58 xfailed (same 5 pre-existing DynamoDB-unavailable errors as baseline); all four backend conformance params green against DynamoDB local + dockerized PG. |
… PG migration 000019)
Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment
artifacts for the V2 schema, pinned to the backend definitions.
- create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables()
wired into create_tables() (always created, like the job queue). The
schemas are an INLINED copy because the script runs standalone in the
dynamodb-init container (bare python + boto3, cannot import
delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts
from delphi_storage.backends.dynamodb.table_schemas("Delphi2_").
- server/postgres/migrations/000019_create_delphi_storage.sql: schema
`delphi` with runs / latest / run_inputs / artifacts / topic_moderation /
collective_statements, mirroring
delphi_storage.backends.postgres.schema_ddl("delphi") statement for
statement (drift-tested the same way).
- tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the
ensure_* test helpers) yield conformant stores: the migration applied by
raw SQL to a scratch database passes the full conformance suite with
ensure_schema off, and script-created Delphi2_* tables pass it with
ensure_tables off. Service-gated tests follow the skip-locally /
fail-in-GitHub-Actions convention.
Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG
all 6 DDL tests pass.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1
(Foundation); Stack 2 (provenance plumbing, P5-P8) follows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d87422d to
328f1de
Compare
… PG migration 000019)
Implements Stack 1 / P4 of docs/STORAGE_V2_DESIGN.md: the deployment
artifacts for the V2 schema, pinned to the backend definitions.
- create_dynamodb_tables.py: DELPHI2_TABLE_SCHEMAS + create_delphi2_tables()
wired into create_tables() (always created, like the job queue). The
schemas are an INLINED copy because the script runs standalone in the
dynamodb-init container (bare python + boto3, cannot import
delphi_storage); tests/test_delphi_storage_ddl.py fails if the copy drifts
from delphi_storage.backends.dynamodb.table_schemas("Delphi2_").
- server/postgres/migrations/000019_create_delphi_storage.sql: schema
`delphi` with runs / latest / run_inputs / artifacts / topic_moderation /
collective_statements, mirroring
delphi_storage.backends.postgres.schema_ddl("delphi") statement for
statement (drift-tested the same way).
- tests/test_delphi_storage_ddl.py proves the DEPLOYED artifacts (not the
ensure_* test helpers) yield conformant stores: the migration applied by
raw SQL to a scratch database passes the full conformance suite with
ensure_schema off, and script-created Delphi2_* tables pass it with
ensure_tables off. Service-gated tests follow the skip-locally /
fail-in-GitHub-Actions convention.
Tests: 406 passed, 30 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); with DynamoDB local + dockerized PG
all 6 DDL tests pass.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 this commit. Completes Stack 1
(Foundation); Stack 2 (provenance plumbing, P5-P8) follows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… pipeline entry points Implements Stack 2 / P5 of docs/STORAGE_V2_DESIGN.md (§4.4): the pipeline job id travels ON THE COMMAND LINE instead of the env-var-only propagation that reached 2 of 18 tables (§3.2). - delphi_storage/job_id.py — resolve_job_id: explicit CLI value > DELPHI_JOB_ID env (transition fallback, removed in a later phase) > auto local-<uuid4> for dev runs. - run_delphi.py — gains --job-id, resolves once, exports DELPHI_JOB_ID for un-migrated env readers (same id as the command lines), and appends --job-id=<id> to all six stage subprocesses (reset, math, umap 500s, 501, 502, 700 per layer). - scripts/job_poller.py — command construction extracted into module-level build_job_command() (now unit-testable); FULL_PIPELINE and CREATE_NARRATIVE_BATCH commands carry --job-id=<queue job_id>; the DELPHI_JOB_ID env export stays for the transition window. 803's existing --job-id (the batch-tracking id, a distinct concept) is unchanged. - Stage entry points all accept --job-id: reset_conversation (CLI moved into cli(), main() stays the worker), run_math_pipeline (arg + log only — no math-logic change), run_pipeline (threaded into process_conversation, replacing the buried env read; auto prefix changes pipeline_run_<uuid> → local-<uuid> for bare dev runs), 501, 502 (arg + log, used from P7), 700 (threaded down to the S3-key builder; 'unknown' fallback preserved), 801 (arg > env, NO auto-generation — narrative section keys embed the id, so a missing one must keep failing loudly downstream). Tests (TDD; tests/test_job_id_threading.py, RED first): - resolve_job_id precedence; run_delphi threads one shared id to all six stage commands (explicit, auto, and env-fallback variants); - every stage entry point's --help advertises --job-id; - process_conversation exposes the job_id parameter; - poller commands for all three job types (FULL_PIPELINE + 801 carry the queue job id; 803 keeps batch_job_id semantics). Full suite: 424 passed, 30 skipped, 58 xfailed (5 pre-existing DynamoDB-unavailable errors unchanged); golden snapshots ran and passed — math outputs untouched. Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
328f1de to
fd5037e
Compare
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + purge tool
Implements the capture half of Stack 2 / P6 (docs/STORAGE_V2_DESIGN.md §4.2
entity 2, §4.4): reproducibility begins with recording EXACTLY what the
pipeline read. The read seam (--input-source, P6b) follows.
- delphi_storage/inputs.py — capture_run_inputs(store, job_id, zid, rid):
writes the six snapshot kinds as run_inputs items (codec envelopes,
zstd-compressed votes) and returns per-kind fingerprints (sha256 of the
canonical uncompressed payload, row_count, max vote created — the P7
manifest inputs). Key fidelity decisions, pinned by tests:
- votes = the RAW stream stage 1 consumes: ORDER BY created, raw PG signs
(the production math path never flips — fetch_votes()'s flip is dead
code), superseded votes included, order preserved (KMeans init depends
on vote-encounter order, INVESTIGATION_K_DIVERGENCE.md).
- The vote SQL is shared BY CONSTRUCTION: run_math_pipeline.py now imports
VOTES_COUNT_SQL / VOTES_BATCH_SQL from delphi_storage.inputs (strings
unchanged; golden suite + the e2e SQL pins stay green).
- comments/participants captured in FULL, including moderated-out rows
(mod = -1) and banned participants — filtering is pipeline behavior,
not snapshot behavior.
- clojure_math_main captured for ALL math_env rows (501 reads env-less
latest, 801 filters MATH_ENV with a different default — snapshotting
every env keeps both replayable).
- derive_votes_latest_unique() reproduces PG's RULE-maintained
votes_latest_unique from the stream (last vote per (pid, tid) in stream
order) — tested against a seeded PG table mimicking the RULE.
- delphi_storage/purge.py — purge_job(): deletes every generic-entity
partition for a job (the §9 GDPR path for snapshot copies).
- run_delphi.py — --snapshot-inputs flag (or DELPHI_SNAPSHOT_INPUTS=1):
captures before any stage runs, aborts the run on capture failure
(a run without recorded inputs defeats the point when enabled).
Off by default until P7's DELPHI_WRITE_MODE subsumes it.
Tests (TDD; tests/test_input_snapshots.py, RED first): 16 tests — raw
stream order/signs, compression, fingerprints, derivation-vs-PG-RULE,
full-row comments/participants, all-env math_main, SQL-parity-by-import,
purge, and the run_delphi hook (flag, env, off-by-default). PG-backed tests
use a seeded scratch database (skip-locally / fail-in-CI convention).
Full suite: 429 passed, 41 skipped, 58 xfailed (5 pre-existing
DynamoDB-unavailable errors unchanged); golden snapshots ran and passed —
math outputs untouched.
Stack: P1 #2597, P2 #2598, P3 #2599, P4 #2600, P5 #2601, P6a this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Introduces Delphi Storage V2 as a backend-neutral Python repository (delphi_storage) with DynamoDB, PostgreSQL, and in-memory backends, plus a shared JSON conformance suite (operation-script case files) intended to match the upcoming TypeScript implementation.
Changes:
- Adds
delphi/delphi_storage/implementing the V2 interface, codec, keying rules, and three backends (memory/DynamoDB/Postgres). - Adds a pytest conformance runner that executes shared JSON case files against all backends (plus backend-specific DynamoDB chunking tests).
- Updates packaging/build artifacts (Hatch wheel packages, Docker copy step) and adds the
zstandarddependency + lock pin.
Reviewed changes
Copilot reviewed 25 out of 28 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| delphi/tests/test_delphi_storage_keys.py | Unit tests for key builders (claim ordering, timestamp format, scopes, artifact keys). |
| delphi/tests/test_delphi_storage_dynamodb.py | DynamoDB-specific tests for crash-safe chunked blob write behavior (moto). |
| delphi/tests/test_delphi_storage_conformance.py | Pytest harness to run shared JSON operation-script conformance cases across backends (incl. race test). |
| delphi/tests/test_delphi_storage_codec.py | Unit tests for canonical JSON, f64 packing, zstd compression, and payload envelopes. |
| delphi/requirements.lock | Pins zstandard in the locked dependency set. |
| delphi/pyproject.toml | Adds zstandard dependency and includes delphi_storage in wheel/sdist packaging. |
| delphi/Dockerfile | Copies delphi_storage/ into builder and runtime images. |
| delphi/delphi_storage/init.py | Exposes the public get_store entrypoint and core models/errors. |
| delphi/delphi_storage/models.py | Defines Pydantic models for manifests, pointers, and KV items. |
| delphi/delphi_storage/keys.py | Defines canonical timestamp validation and key/scope builders (cross-language contract). |
| delphi/delphi_storage/interface.py | Defines the backend-neutral repository interface and shared contract validation helpers. |
| delphi/delphi_storage/factory.py | Implements backend selection via env vars (`dynamodb |
| delphi/delphi_storage/codec.py | Implements canonical JSON + zstd + packed-f64 payload envelopes and DynamoDB number helpers. |
| delphi/delphi_storage/backends/init.py | Backend package marker. |
| delphi/delphi_storage/backends/memory.py | In-memory reference backend used as an executable spec + test double. |
| delphi/delphi_storage/backends/dynamodb.py | DynamoDB backend including chunked blob storage, optimistic locking, GSIs, and latest pointers. |
| delphi/delphi_storage/backends/postgres.py | PostgreSQL backend with schema-scoped tables, COLLATE "C" ordering, and SKIP LOCKED claims. |
| delphi/delphi_storage/conformance/init.py | Conformance package marker. |
| delphi/delphi_storage/conformance/README.md | Normative conformance spec describing entities, operations, ordering, and case format. |
| delphi/delphi_storage/conformance/runner.py | JSON case loader/executor and assertion utilities for conformance execution. |
| delphi/delphi_storage/conformance/cases/roundtrip_basic.json | Case: basic put/get round-trips, unicode, numbers-by-value, overwrite. |
| delphi/delphi_storage/conformance/cases/query_ordering.json | Case: UTF-8 byte-order prefix/between queries + delete_partition behavior. |
| delphi/delphi_storage/conformance/cases/blob_chunking.json | Case: large blob round-trips + chunk invisibility + overwrite shrink cleanup. |
| delphi/delphi_storage/conformance/cases/queue_claim.json | Case: claim ordering, leases, duplicate enqueue, terminal transitions, list_runs. |
| delphi/delphi_storage/conformance/cases/latest_pointer.json | Case: monotonic latest seq, idempotent completion, importer no-clobber invariant, rid scopes. |
| delphi/delphi_storage/conformance/cases/run_manifest_fields.json | Case: merge_run_fields provenance accumulation + append_log sequencing as artifacts. |
| delphi/delphi_storage/conformance/cases/validation.json | Case: invalid writes rejected + runs/latest unreachable via generic ops + status/type validation. |
| delphi/delphi_storage/conformance/cases/codec_fixtures.json | Case: cross-language codec fixtures and round-trips for envelope encodings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if enc == "json+zstd": | ||
| return json.loads(raw.decode("utf-8")) | ||
| return unpack_f64(raw) |
| def _numbers_equal(a: Any, b: Any) -> bool: | ||
| return not isinstance(a, bool) and not isinstance(b, bool) and float(a) == float(b) |
| import uuid | ||
|
|
||
| import pytest | ||
|
|
||
| moto = pytest.importorskip("moto") | ||
|
|
||
| from delphi_storage.backends.dynamodb import ( # noqa: E402 - after importorskip | ||
| CHUNK_SIZE, | ||
| DynamoDelphiStore, | ||
| _chunk_prefix, | ||
| _chunk_sk, | ||
| ) | ||
| from delphi_storage.models import StoreItem # noqa: E402 | ||
|
|
||
|
|
||
| @pytest.fixture() | ||
| def store(): | ||
| with moto.mock_aws(): | ||
| yield DynamoDelphiStore( | ||
| table_prefix=f"ChunkT{uuid.uuid4().hex[:8]}_", | ||
| region="us-east-1", | ||
| ensure_tables=True, | ||
| ) |
| rows = self._query_all( | ||
| TableName=self._runs_table, | ||
| IndexName=index, | ||
| KeyConditionExpression=f"{attr} = :key", | ||
| ExpressionAttributeValues={":key": {"S": value}}, |
fd5037e to
3c07e95
Compare
Implements Stack 1 / P2 of docs/STORAGE_V2_DESIGN.md (§4.2-4.3): the neutral repository interface over the six logical entities (runs, run_inputs, artifacts, latest, topic_moderation, collective_statements), with BOTH production backends and the shared conformance spec that keeps the Python and TypeScript implementations honest. - delphi_storage/interface.py — DelphiStore ABC: generic put/get/ query_prefix/query_between/delete_partition + semantic queue ops (enqueue_run, claim_next_run, extend_lease, update_run_status, merge_run_fields, complete_run, append_log, advance_latest, get_latest, list_runs). complete_run flips the latest pointer last (commit point), idempotent and crash-healing; advance_latest carries the importer no-clobber mode (§6.2 invariant 1). - delphi_storage/codec.py — canonical JSON, little-endian packed float64, zstd, payload envelopes (json / json+zstd / f64+zstd; sha256 + byte counts of the uncompressed payload), Decimal conversion for DynamoDB. - delphi_storage/keys.py — claim-order strings (priority desc then FIFO), canonical millisecond-UTC timestamps, latest scopes, log keys. - backends/dynamodb.py — thread-safe low-level client, conditional writes on version (lift of the job_poller optimistic lock), sparse claim GSI, transparent >300KB blob chunking behind the interface. - backends/postgres.py — schema-scoped tables, COLLATE "C" byte-order keys, FOR UPDATE SKIP LOCKED claims; DDL is the single Python-side source that migration 000019 (P4) will mirror. - backends/memory.py — in-process reference implementation. - conformance/ — normative README + 8 shared JSON case files executed by pytest across ALL backends (memory always; moto always; real DynamoDB and PostgreSQL under the skip-locally/fail-in-CI convention), covering round-trips (unicode/floats/blobs), UTF-8 byte-order queries, chunk-spanning payloads, queue claim ordering + leases, monotonic latest pointers, idempotent completion, importer no-clobber, and validation. A threaded two-claimants-one-wins race test runs against memory + both real backends. - Dockerfile — add delphi_storage/ to the builder and final COPY layers, the same way the sibling first-party packages polismath/ and umap_narrative/ are baked into the image. Without it the image builds fine but `import delphi_storage` raises ModuleNotFoundError at pytest collection in the CI test stage (the package is declared in pyproject but its source was never copied into the image). - backends/postgres.py — normalize the legacy `postgres://` URL scheme to `postgresql://` in PostgresDelphiStore. SQLAlchemy 2.0 dropped the `postgres://` alias and raises NoSuchModuleError on it, but that's still what many platforms (Heroku et al.) put in DATABASE_URL — so a production DATABASE_URL would otherwise crash engine creation. Only the scheme is rewritten; any +driver suffix is preserved. Unit-tested without a live DB (tests/test_delphi_storage_postgres.py; create_engine resolves the dialect eagerly but opens no connection). Tests: 398 passed, 28 skipped, 58 xfailed (baseline: 332/12/58; the 5 pre-existing DynamoDB-unavailable errors unchanged). With DynamoDB local (:8002) + dockerized PG: all 4 backends fully green. Part of the storage-v2 stack (P1 = #2597, the design doc). P3 (TypeScript twin) and P4 (DDL) follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3c07e95 to
afbe110
Compare
Delphi Coverage Report
|
Stack 1 / P2 of the Storage V2 effort — design doc in #2597 (
delphi/docs/STORAGE_V2_DESIGN.md, §4.2–4.3, §7).What this adds
A new Python package
delphi/delphi_storage/: the backend-neutral repository over the six V2 logical entities (runs,run_inputs,artifacts,latest,topic_moderation,collective_statements), with both production backends and the shared conformance spec that will keep the Python and TypeScript implementations honest.Interface (
interface.py)put / put_batch / get / query_prefix / query_between / delete_partition— only for the four KV entities;runs/latestare reachable exclusively through semantic ops so their invariants can't be bypassed.enqueue_run,claim_next_run(priority desc → FIFO → job_id, conditional-write atomicity),extend_lease,update_run_status,merge_run_fields,complete_run,append_log,advance_latest,get_latest,list_runs.complete_runwrites thelatestpointer last (the commit point, design §4.2), is idempotent, and heals a crash between manifest-COMPLETED and pointer flip.advance_latest(only_if_absent_or_imported=True)implements the backfill importer's no-clobber invariant (§6.2 invariant 1) — conformance-tested now, used by P9.Codec (
codec.py)Canonical JSON (sorted keys, compact, UTF-8, NaN rejected), little-endian packed float64 (bit-exact), zstd, and versioned payload envelopes (
json/json+zstd/f64+zstd) carrying sha256 + byte counts of the uncompressed payload (compressed bytes are never compared: zstd output is implementation-dependent). Decimal↔float conversion for DynamoDB follows the existingDecimal(str(x))convention.Backends
backends/dynamodb.py): thread-safe low-level client + TypeSerializer; optimistic-lock conditional writes onversion(the lift of the workingjob_poller.pyclaim logic); sparseclaim-indexGSI for queue ordering;zid-index/rid-indexforlist_runs; blobs >300KB transparently chunked across hidden rows (sk prefixed U+007F) and reassembled on read — invisible to queries, stale chunks cleaned on overwrite.backends/postgres.py): schema-scoped tables (defaultdelphi),COLLATE "C"sort keys (= DynamoDB byte order),FOR UPDATE SKIP LOCKEDclaims, JSONB manifests following the legacymath_mainpattern. The DDL in this file is the single Python-side source that migration000019(P4) mirrors.backends/memory.py): in-process reference implementation / executable spec.factory.py:DELPHI_STORAGE_BACKEND=dynamodb|postgres|memory(+DELPHI_STORAGE_TABLE_PREFIX,DELPHI_STORAGE_PG_SCHEMA,DELPHI_STORAGE_PG_URL), reusing the existingDYNAMODB_ENDPOINT/AWS_REGION/DATABASE_URLvocabulary.Conformance suite (
conformance/)conformance/README.md+ 8 JSON operation-script case files — the same files P3's jest suite executes against the TS backends.memoryandmotoalways; realdynamodb/postgresskip locally when unreachable and fail loudly underGITHUB_ACTIONS(both services are up in CI).Testing
:8002) + dockerized PG 16: all backend×case params green, including the race tests.Dependencies
Adds
zstandard>=0.23topyproject.tomland the matching pin torequirements.lock(Docker build).Stack
server/src/storage/delphi/+ jest on the same case files (next)create_dynamodb_tables.pyadditions + PG migration000019🤖 Generated with Claude Code