From 97646c48bab5863e64f3d141961f4c474b86fc56 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 16:30:55 +0300 Subject: [PATCH 01/14] Plan --- .../2026-07-14-detext-equal-branch-filter.md | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/plans/2026-07-14-detext-equal-branch-filter.md diff --git a/docs/plans/2026-07-14-detext-equal-branch-filter.md b/docs/plans/2026-07-14-detext-equal-branch-filter.md new file mode 100644 index 0000000..b14e250 --- /dev/null +++ b/docs/plans/2026-07-14-detext-equal-branch-filter.md @@ -0,0 +1,119 @@ +# Handoff: filtering `fix-collision-stuff/2` down to what survives without `Text/equal` + +## Why this exists + +`fix-collision-stuff/2` accumulated 15 commits (all by Viacheslav Shvets) +since base commit `7b876a3e4f80d764ec16eff802288ec28ccda920`, several of +which lean on `Text/equal` — an experimental, pgn-fork-only Dhall builtin +that is going away in a future pgn/Dhall release. Stock Dhall has no text +equality primitive at all (a deliberate upstream omission), so any code +depending on `Text/equal` needs to be either eliminated or explicitly +accepted as blocked debt. This document is the outcome of a full audit of +which parts survive and which don't, and why. + +**Status when this was written:** decisions are locked (see table below). +Implementation has **not** started. Nikita is going to attempt a +gen-contract change first (see item 4) before any code gets written here, +since that changes what "keep, flagged as blocked" for `buildLookup` should +actually look like. + +## Decision table + +| # | Item | Disposition | +|---|---|---| +| 1 | `emitSync` — dual sync+async generation in one module (`e46dc96`, `a1ac9f8`, `e7d1e6b`) | **Keep as-is.** Confirmed zero `Text/equal` dependency. | +| 2 | Custom type registration — psycopg-native adapter binding, dependency-ordered via a `Natural` `order` field (`bac8f95`, `375ba6f`) | **Keep as-is.** The registration/ordering mechanism itself is zero `Text/equal` (`sameOrder` uses `Natural/subtract`, not text comparison). | +| 3 | `50f74fa`'s module-internal reserved-name disambiguation (`querySafeName`, `moduleReservedNames`, `parameterSafeName` in `PyIdent.dhall`) | **Keep as-is.** Necessary regardless of any collision-detection decision (stops e.g. a query named `date` from shadowing its own generated file's `from datetime import date`), and already `Text/equal`-free via the same fixed-list `Text/replace` trick as Python-keyword escaping. | +| 4 | `buildLookup` (`5147204`, `Interpreters/Project.dhall:590-611`) — backs `onUnsupported: Skip`'s cascade removal and the empirically-verified rank-limit validation (composite arrays ≤1 dim, enum arrays ≤2 dims) | **Keep, flagged as blocked debt** — the sole accepted `Text/equal` exception, pending a gen-contract change. **Nikita is pursuing this now** (see "What #4 needs" below). | +| 5 | Flat top-level package facade (re-exports every query/type/core symbol into one shared top-level namespace) | **Replace.** Un-flatten into per-kind sub-namespaces (statements get their own namespace, types get their own). This is new work, not a keep/discard of an existing commit. | +| 6 | Same-kind collision *detection* (two queries, or two custom types, whose generated names collide with each other — as opposed to cross-kind collisions, which item 5 eliminates structurally) | **Discard.** Risk accepted; pushed upstream instead. Filed [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asking pgn to reject non-normalized query filenames and guarantee unique custom-type identities at the source. | +| 7 | Manual rename-override feature (`queryNameMappings`/`customTypeNameMappings` config, `Structures/PythonNameMapping.dhall`) | **Discard entirely.** Wasn't on the original keep-list; not needed once detection (item 6) is dropped; was the last thing in the collision cluster still needing `Text/equal` (to match a reference's name against a configured mapping's source). | +| 8 | `docs/adr/0001-generated-python-name-collisions.md` and related docs (`DESIGN.md`, `README.md`, `CHANGELOG.md` sections describing the flat facade / collision detection / rename mappings) | **Drop the ADR outright** (decided 2026-07-14, superseding the earlier "rewrite" plan). Related doc sections describing the now-discarded flat-facade/detection/rename-mapping behavior still need rewriting or removal to match the new state — not a blanket drop, just the ADR itself. | +| — | Tests exercising discarded behavior (`test_python_name_collisions.py`, `test_takeover_contract.py`) | Rewrite/remove to match — `test_python_name_collisions.py` specifically tests the collision-detection and rename-mapping mechanisms being discarded; `test_takeover_contract.py` locks in flat-facade output that item 5 changes. | + +## What #4 (buildLookup) needs from gen-contract + +The generator's `Model.Scalar.Custom` variant (pinned to gen-contract +v4.0.1) carries only a bare `Name`: + +```dhall +let ScalarDecode = < Passthrough | Custom > +-- Custom = \(name : Model.Name) -> { customRef = Some name, decode = ScalarDecode.Custom, ... } +``` + +There's no id or kind tag tying a *reference* to a custom type back to its +*definition* without a project-wide name search — that search is what +`buildLookup` performs with `Text/equal`. The fix already drafted (and +previously withdrawn, then un-withdrawn — see `docs/upstream-asks.md` ask +3 in this branch's history) is to add a stable identity directly onto +`Scalar.Custom`: + +``` +Scalar.Custom : { name : Name, kind : < Enum | Composite >, id : Natural } +``` + +If the reference site already carries this, `buildLookup`, +`Structures/CustomKind.dhall`, and the `Text/equal` dependency disappear +from python.gen entirely — the classification comes for free instead of +being re-derived by search. This touches every gen-sdk consumer (java.gen +included), so it needs coordination, not something python.gen can do +unilaterally against a pinned import. + +Full background, live evidence (reproduced `pgn generate` runs showing the +Skip-cascade and rank-limit behavior in detail), and the historical context +of why `buildLookup` was withdrawn-then-reinstated live in the companion +note: `/private/tmp/claude-501/-Users-mojojojo-repos-pgenie-python-gen/018e3c75-72c2-414c-af89-0e300db2f020/scratchpad/handoff-buildlookup-text-equal.md` +(session-local scratch path — copy anything worth keeping into this repo +before that path is cleaned up). + +**Once the gen-contract change lands and this branch's pin is bumped:** +`buildLookup` should be reimplemented against the new `id`/`kind` fields +directly (no search, no `Text/equal`) rather than kept as flagged debt — +re-open this document and update item 4's disposition before starting +implementation. + +## What the facade un-flattening (item 5) needs to look like + +Not yet designed. The current flat facade +(`tests/golden/src/specimen_client/__init__.py`) does e.g.: + +```python +from ._generated._core import JsonValue as JsonValue +from ._generated.statements.get_specimen import GetSpecimenRow as GetSpecimenRow +``` + +— everything at one flat top level. The replacement needs statements and +types each importable from their own sub-namespace instead, so that e.g. a +custom type named `json_value` and the reserved core symbol `JsonValue` +never land in the same importable namespace, and a query's `FooRow` and an +unrelated custom type also named `foo_row` don't either. Exact shape +(e.g. `from mypackage.statements import get_specimen` / +`from mypackage.types import Mood` vs. some other split) still needs +deciding — this was flagged during the grill as needing design work, not +resolved in detail. + +Note this interacts with the "takeover-ready client" ergonomics work +(`1a5de19`, `ae2869c`) which specifically aimed for a flat, single-import +surface — un-flattening is a deliberate reversal of that goal, traded for +structural collision-safety. `5ac6d49`'s `test_takeover_contract.py` and +`a1ac9f8`'s Ruff-ordering fix both assume the flat shape and will need +rework. + +## What's already done, independent of the above + +- **[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75)** + filed, asking pgn to reject non-normalized query filenames (confirmed + live: `get-specimen.sql` and `get_specimen.sql` both resolve to + `get_specimen`, hyphenated filenames are accepted by pgn on their own — + not rejected upstream) and guarantee unique custom-type identities at the + source. + +## Next steps + +1. Nikita attempts the gen-contract change for item 4. +2. Come back to this document, update item 4's disposition based on the + outcome (clean reimplementation vs. still-blocked debt). +3. Only then start implementation: new commits on top of current `HEAD` + (not a history rewrite — the 15 commits' hypothesis-testing context in + their messages is worth keeping), covering items 5-8 plus whatever + item 4 turns into. From eb242514ad20863d05ff06b133c50d8a437d69fc Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 19:33:14 +0300 Subject: [PATCH 02/14] Prohibit local cross-repo deps --- AGENTS.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 7da3b7b..79f0a1f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,3 +36,9 @@ already says. `src/` pins its remote imports by sha256 (`src/Deps/*.dhall`). Bump those deliberately, one at a time, and re-run the harness before committing a pin change. + +Never replace a pin with a local filesystem path (`../gen-contract/...`, +`../gen-sdk/...`) even temporarily for convenience; it only works on a +machine with those sibling repos checked out and breaks CI. Always use a +`https://raw.githubusercontent.com/pgenie-io///...` import with +its `sha256`. From 9f8284917fba7992b88ccab6b76a3e2be3a2435f Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 19:33:25 +0300 Subject: [PATCH 03/14] Update the plan --- .../2026-07-14-detext-equal-branch-filter.md | 151 +++++++++++++----- 1 file changed, 113 insertions(+), 38 deletions(-) diff --git a/docs/plans/2026-07-14-detext-equal-branch-filter.md b/docs/plans/2026-07-14-detext-equal-branch-filter.md index b14e250..e05edfd 100644 --- a/docs/plans/2026-07-14-detext-equal-branch-filter.md +++ b/docs/plans/2026-07-14-detext-equal-branch-filter.md @@ -12,11 +12,19 @@ accepted as blocked debt. This document is the outcome of a full audit of which parts survive and which don't, and why. **Status when this was written:** decisions are locked (see table below). -Implementation has **not** started. Nikita is going to attempt a +Implementation has **not** started. Nikita was going to attempt a gen-contract change first (see item 4) before any code gets written here, since that changes what "keep, flagged as blocked" for `buildLookup` should actually look like. +**Update 2026-07-14 (later the same day):** the gen-contract change +happened and shipped — `gen-contract` v5.0.0 and `gen-sdk` v3.0.0 are now +released. Item 4 is no longer blocked debt; see its updated row and the +rewritten "What #4 needs" section below (now "What #4 gets"). This branch's +pins (`src/Deps/Contract.dhall` at v4.0.1, `src/Deps/Sdk.dhall` at v2.0.0) +have **not** been bumped yet — that bump plus the resulting migration is +now the actual next step, not a hypothetical. + ## Decision table | # | Item | Disposition | @@ -24,53 +32,106 @@ actually look like. | 1 | `emitSync` — dual sync+async generation in one module (`e46dc96`, `a1ac9f8`, `e7d1e6b`) | **Keep as-is.** Confirmed zero `Text/equal` dependency. | | 2 | Custom type registration — psycopg-native adapter binding, dependency-ordered via a `Natural` `order` field (`bac8f95`, `375ba6f`) | **Keep as-is.** The registration/ordering mechanism itself is zero `Text/equal` (`sameOrder` uses `Natural/subtract`, not text comparison). | | 3 | `50f74fa`'s module-internal reserved-name disambiguation (`querySafeName`, `moduleReservedNames`, `parameterSafeName` in `PyIdent.dhall`) | **Keep as-is.** Necessary regardless of any collision-detection decision (stops e.g. a query named `date` from shadowing its own generated file's `from datetime import date`), and already `Text/equal`-free via the same fixed-list `Text/replace` trick as Python-keyword escaping. | -| 4 | `buildLookup` (`5147204`, `Interpreters/Project.dhall:590-611`) — backs `onUnsupported: Skip`'s cascade removal and the empirically-verified rank-limit validation (composite arrays ≤1 dim, enum arrays ≤2 dims) | **Keep, flagged as blocked debt** — the sole accepted `Text/equal` exception, pending a gen-contract change. **Nikita is pursuing this now** (see "What #4 needs" below). | +| 4 | `buildLookup` (`5147204`, `Interpreters/Project.dhall:590-611`) — backs `onUnsupported: Skip`'s cascade removal and the empirically-verified rank-limit validation (composite arrays ≤1 dim, enum arrays ≤2 dims) | **Resolved, not yet implemented.** `gen-contract` v5.0.0 + `gen-sdk` v3.0.0 (released 2026-07-14) eliminate the need for `buildLookup`, `Text/equal`, and the hand-rolled `CustomKind.Lookup`/`Natural/fold` cascade entirely — see "What #4 gets" below. This branch's pins are still on v4.0.1/v2.0.0; bumping them and migrating `Interpreters/Project.dhall`/`Scalar.dhall`/`Structures/CustomKind.dhall` is now real work, not a keep-as-debt decision. | | 5 | Flat top-level package facade (re-exports every query/type/core symbol into one shared top-level namespace) | **Replace.** Un-flatten into per-kind sub-namespaces (statements get their own namespace, types get their own). This is new work, not a keep/discard of an existing commit. | | 6 | Same-kind collision *detection* (two queries, or two custom types, whose generated names collide with each other — as opposed to cross-kind collisions, which item 5 eliminates structurally) | **Discard.** Risk accepted; pushed upstream instead. Filed [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asking pgn to reject non-normalized query filenames and guarantee unique custom-type identities at the source. | | 7 | Manual rename-override feature (`queryNameMappings`/`customTypeNameMappings` config, `Structures/PythonNameMapping.dhall`) | **Discard entirely.** Wasn't on the original keep-list; not needed once detection (item 6) is dropped; was the last thing in the collision cluster still needing `Text/equal` (to match a reference's name against a configured mapping's source). | | 8 | `docs/adr/0001-generated-python-name-collisions.md` and related docs (`DESIGN.md`, `README.md`, `CHANGELOG.md` sections describing the flat facade / collision detection / rename mappings) | **Drop the ADR outright** (decided 2026-07-14, superseding the earlier "rewrite" plan). Related doc sections describing the now-discarded flat-facade/detection/rename-mapping behavior still need rewriting or removal to match the new state — not a blanket drop, just the ADR itself. | | — | Tests exercising discarded behavior (`test_python_name_collisions.py`, `test_takeover_contract.py`) | Rewrite/remove to match — `test_python_name_collisions.py` specifically tests the collision-detection and rename-mapping mechanisms being discarded; `test_takeover_contract.py` locks in flat-facade output that item 5 changes. | -## What #4 (buildLookup) needs from gen-contract +## What #4 (buildLookup) gets from gen-contract v5.0.0 / gen-sdk v3.0.0 + +Both shipped 2026-07-14. What landed is better than the `id`/`kind` +addition originally drafted here: `gen-sdk` ships a ready-made, +`Text/equal`-free replacement for the whole cascade, not just the +identity tag. -The generator's `Model.Scalar.Custom` variant (pinned to gen-contract -v4.0.1) carries only a bare `Name`: +**gen-contract v5.0.0** (`Scalar.Custom` breaking change, commit +`e10128f`): the bare `Name` becomes a resolvable `CustomTypeRef`: ```dhall -let ScalarDecode = < Passthrough | Custom > --- Custom = \(name : Model.Name) -> { customRef = Some name, decode = ScalarDecode.Custom, ... } +let CustomTypeRef = + { name : Name, pgSchema : Text, pgName : Text, index : Natural } + +let Scalar = < Primitive : Primitive | Custom : CustomTypeRef > ``` -There's no id or kind tag tying a *reference* to a custom type back to its -*definition* without a project-wide name search — that search is what -`buildLookup` performs with `Text/equal`. The fix already drafted (and -previously withdrawn, then un-withdrawn — see `docs/upstream-asks.md` ask -3 in this branch's history) is to add a stable identity directly onto -`Scalar.Custom`: +`index` points into `Project.customTypes : List CustomType`, and the +contract now guarantees that list is **topologically sorted**: every index +reachable from `customTypes[i]` is `< i`. So a reference resolves via +`Prelude.List.index ref.index _ project.customTypes` — a Natural-indexed +list lookup — never a name search, never `Text/equal`. Getting the +referenced type's *kind* (Composite/Enum/Domain) is then a plain union +`merge` on `.definition`, also `Text/equal`-free. + +**gen-sdk v3.0.0** (commit `bba067e` et al., pinned to gen-contract v5.0.0): +adds a `CustomTypes` module built on exactly one topological left fold over +`Project.customTypes`, exposing: + +- `supportedCustomTypes : (CustomTypeDefinition -> Bool) -> List CustomType -> List Bool` — + per-type supported/unsupported, propagated through nested + Composite/Domain dependencies via index lookups into the fold's own + running output (no re-traversal, no `Natural/fold`-over-`List/filter` + cascade like `effectiveResolvedCustomTypes` currently does). +- `supportedCustomTypesReasoned` — same fold, but returns the *root-cause* + `CustomTypeRef` for each unsupported type instead of a bare `Bool` + (useful for warning messages). +- `customTypeIsSupported : List Bool -> CustomTypeRef -> Bool` and + `queryIsSupported : List Bool -> Query -> Bool` — given the fold's output, + answer whether a specific reference or an entire query (all its params + and result columns, transitively) survives. + +This directly supersedes python.gen's own `buildLookup` + +`effectiveResolvedCustomTypes` cascade-removal loop +(`Interpreters/Project.dhall:1174-1204`), which was reinventing the same +topological propagation with repeated `Natural/fold` passes over +`Text/equal`-keyed lookups because the input wasn't guaranteed sorted +before. It does **not** replace the rank-limit checks in +`ParamsMember.dhall:294/318` and `Member.dhall:92/109` (array of enum >2 +dims / composite >1 dim) — those are python.gen-specific and still need to +know a referenced type's kind, but can now get it directly off +`ref.index` (via a `List CustomKind.TypeKind` built once, index-aligned +with `project.customTypes`, mirroring gen-sdk's own `List Bool` pattern) +instead of through `Structures/CustomKind.dhall`'s `Model.Name -> TypeKind` +closure and its `Text/equal`-based construction in `buildLookup`. + +Net effect once migrated: `buildLookup`, `Structures/CustomKind.dhall`'s +`Lookup` type, and the `Text/equal` call at +`Interpreters/Project.dhall:601` all go away. `Structures/CustomKind.dhall` +likely still keeps `TypeKind`/`Identity` (renamed/reshaped to be +index-keyed) since `CustomTypeGen.run` and the Python codegen side still +need the className/moduleName/order identity — only the *lookup mechanism* +is replaced, not the whole module. + +Relevant pins for the migration (computed via `dhall hash`, not raw file +`shasum` — Dhall pins are semantic-CBOR hashes, not byte hashes): ``` -Scalar.Custom : { name : Name, kind : < Enum | Composite >, id : Natural } +-- src/Deps/Contract.dhall +https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall + sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef + +-- src/Deps/Sdk.dhall +https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall + sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 ``` -If the reference site already carries this, `buildLookup`, -`Structures/CustomKind.dhall`, and the `Text/equal` dependency disappear -from python.gen entirely — the classification comes for free instead of -being re-derived by search. This touches every gen-sdk consumer (java.gen -included), so it needs coordination, not something python.gen can do -unilaterally against a pinned import. +Both changelogs (`gen-contract` v5.0.0, `gen-sdk` v3.0.0) explicitly call +out the `Scalar.Custom` shape as a **breaking change** — bumping the pin +alone will not compile; every site currently pattern-matching +`Scalar.Custom` with a bare `Name` (`Interpreters/Scalar.dhall:43-51` at +minimum) breaks and needs updating to destructure `CustomTypeRef` instead. +gen-sdk v3.0.0's `Fixtures/Exhaustive.dhall` also grew a domain custom type +and a composite-over-domain type in a non-`public` schema with a +divergent `pgName`, specifically to exercise this — python.gen's own +fixtures/golden tests should be checked against that once the pin bumps. Full background, live evidence (reproduced `pgn generate` runs showing the -Skip-cascade and rank-limit behavior in detail), and the historical context -of why `buildLookup` was withdrawn-then-reinstated live in the companion -note: `/private/tmp/claude-501/-Users-mojojojo-repos-pgenie-python-gen/018e3c75-72c2-414c-af89-0e300db2f020/scratchpad/handoff-buildlookup-text-equal.md` -(session-local scratch path — copy anything worth keeping into this repo -before that path is cleaned up). - -**Once the gen-contract change lands and this branch's pin is bumped:** -`buildLookup` should be reimplemented against the new `id`/`kind` fields -directly (no search, no `Text/equal`) rather than kept as flagged debt — -re-open this document and update item 4's disposition before starting -implementation. +old Skip-cascade and rank-limit behavior in detail), and the historical +context of why `buildLookup` was withdrawn-then-reinstated live in the +companion note: `/private/tmp/claude-501/-Users-mojojojo-repos-pgenie-python-gen/018e3c75-72c2-414c-af89-0e300db2f020/scratchpad/handoff-buildlookup-text-equal.md` +(session-local scratch path, from the *original* audit session — may +already be cleaned up; copy anything worth keeping into this repo). ## What the facade un-flattening (item 5) needs to look like @@ -110,10 +171,24 @@ rework. ## Next steps -1. Nikita attempts the gen-contract change for item 4. -2. Come back to this document, update item 4's disposition based on the - outcome (clean reimplementation vs. still-blocked debt). -3. Only then start implementation: new commits on top of current `HEAD` - (not a history rewrite — the 15 commits' hypothesis-testing context in - their messages is worth keeping), covering items 5-8 plus whatever - item 4 turns into. +1. ~~Nikita attempts the gen-contract change for item 4.~~ Done — shipped + as `gen-contract` v5.0.0 + `gen-sdk` v3.0.0, both released 2026-07-14. +2. ~~Come back to this document, update item 4's disposition.~~ Done above + — clean reimplementation via gen-sdk's new `CustomTypes` module, not + blocked debt. +3. Bump this branch's pins (`src/Deps/Contract.dhall` → v5.0.0, + `src/Deps/Sdk.dhall` → v3.0.0, hashes above) and fix the resulting + compile breakage at every `Scalar.Custom` pattern match + (`Interpreters/Scalar.dhall` at minimum — grep for other sites once the + bump is in). +4. Migrate `buildLookup`/`effectiveResolvedCustomTypes` + (`Interpreters/Project.dhall:590-611,1174-1204`) onto + `Sdk.CustomTypes.{supportedCustomTypesReasoned, queryIsSupported}`, and + reshape `Structures/CustomKind.dhall`'s `Lookup` from a + `Model.Name -> TypeKind` closure to an index-aligned `List TypeKind` + sourced from `ref.index`. Confirm the rank-limit checks in + `ParamsMember.dhall`/`Member.dhall` still get the kind they need through + the new shape. +5. Only then start implementation on items 5-8: new commits on top of + current `HEAD` (not a history rewrite — the 15 commits' + hypothesis-testing context in their messages is worth keeping). From d18398c70406e18462c32528b88d50af59521982 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 19:59:29 +0300 Subject: [PATCH 04/14] Bump gen-contract to v5.0.0 and gen-sdk to v3.0.0 Scalar.Custom now carries a CustomTypeRef (name, pgSchema, pgName, index) instead of a bare Name, and Value flattens its optional array wrapper into plain dimensionality/elementIsNullable fields. Updated every site touched by these two shape changes (Scalar, Value, Member, ParamsMember, CustomType interpreters) to compile against the new contract while preserving current behavior exactly; the buildLookup/CustomKind.Lookup mechanism itself is untouched (that migration is a separate follow-up task). BLOCKED on verification: pgn v0.9.1 (pinned in mise.toml) hard-rejects the new contract's major version before loading the generator at all ("Incompatible contract major version: 5. Expected 4."), so mise run test and mise run golden cannot currently exercise this change end-to-end. See .superpowers/sdd/task-1-report.md for the full failure output and analysis. --- CHANGELOG.md | 5 +++++ src/Deps/Contract.dhall | 4 ++-- src/Deps/Sdk.dhall | 4 ++-- src/Interpreters/CustomType.dhall | 16 +++++-------- src/Interpreters/Member.dhall | 12 +++++----- src/Interpreters/ParamsMember.dhall | 28 +++++++++++------------ src/Interpreters/Scalar.dhall | 10 ++++----- src/Interpreters/Value.dhall | 35 +++++++++++++---------------- 8 files changed, 54 insertions(+), 60 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ea7e441..698d769 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -90,3 +90,8 @@ and wheel are GPL-3.0-or-later because they inline gen-sdk. The wheel includes the sha256-pinned GPL text and `wheel/NOTICE`; PyPI publication remains disabled behind its explicit gate. + +- Bumped gen-contract to v5.0.0 and gen-sdk to v3.0.0. `Scalar.Custom` now + carries a `CustomTypeRef` (name, pgSchema, pgName, index) instead of a bare + `Name`, and `Value` flattens its optional array wrapper into plain + `dimensionality`/`elementIsNullable` fields. diff --git a/src/Deps/Contract.dhall b/src/Deps/Contract.dhall index ea9e805..6fdb9e8 100644 --- a/src/Deps/Contract.dhall +++ b/src/Deps/Contract.dhall @@ -1,2 +1,2 @@ -https://raw.githubusercontent.com/pgenie-io/gen-contract/v4.0.1/src/package.dhall - sha256:4a130ba7fbaa152a776babbb1bf2994a4833931ca76bde9bf6930d354225651e +https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall + sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef diff --git a/src/Deps/Sdk.dhall b/src/Deps/Sdk.dhall index 18f9fce..95de509 100644 --- a/src/Deps/Sdk.dhall +++ b/src/Deps/Sdk.dhall @@ -1,2 +1,2 @@ -https://raw.githubusercontent.com/pgenie-io/gen-sdk/v2.0.0/src/package.dhall - sha256:b9def6ab1179bc4aaae7fc6e91977f094f75934cd5755175c294a9e97ca71b15 +https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall + sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 diff --git a/src/Interpreters/CustomType.dhall b/src/Interpreters/CustomType.dhall index 1b59985..bf0241b 100644 --- a/src/Interpreters/CustomType.dhall +++ b/src/Interpreters/CustomType.dhall @@ -239,18 +239,14 @@ let run = \(_ : Model.Primitive) -> MemberGen.run {=} lookup m , Custom = - \(name : Model.Name) -> - Prelude.Optional.fold - Model.ArraySettings - m.value.arraySettings - (Lude.Compiled.Type MemberGen.Output) - ( \(_ : Model.ArraySettings) -> - Lude.Compiled.report + \(ref : Model.CustomTypeRef) -> + if Prelude.Bool.not + (Natural/isZero m.value.dimensionality) + then Lude.Compiled.report MemberGen.Output - [ m.pgName, name.inSnakeCase ] + [ m.pgName, ref.name.inSnakeCase ] "Custom array fields inside a composite type are not supported" - ) - (MemberGen.run {=} lookup m) + else MemberGen.run {=} lookup m } m.value.scalar ) diff --git a/src/Interpreters/Member.dhall b/src/Interpreters/Member.dhall index c621e6d..bbe05b3 100644 --- a/src/Interpreters/Member.dhall +++ b/src/Interpreters/Member.dhall @@ -49,10 +49,10 @@ let runWithPrefix = } , Custom = Prelude.Optional.fold - Model.Name + Model.CustomTypeRef value.scalar.customRef (Lude.Compiled.Type Output) - ( \(name : Model.Name) -> + ( \(ref : Model.CustomTypeRef) -> let mkOutput = \(identity : CustomKind.Identity) -> \(customImports : ImportSet.Type) -> @@ -87,7 +87,7 @@ let runWithPrefix = else Lude.Compiled.report Output [ input.pgName - , name.inSnakeCase + , ref.name.inSnakeCase ] "Array of an enum with dimensionality > 2 is not supported" , Composite = @@ -104,16 +104,16 @@ let runWithPrefix = else Lude.Compiled.report Output [ input.pgName - , name.inSnakeCase + , ref.name.inSnakeCase ] "Array of a composite type with dimensionality > 1 is not supported" , Absent = Lude.Compiled.report Output - [ name.inSnakeCase ] + [ ref.name.inSnakeCase ] "Custom type not found in project customTypes" } - (lookup name) + (lookup ref.name) ) ( Lude.Compiled.report Output diff --git a/src/Interpreters/ParamsMember.dhall b/src/Interpreters/ParamsMember.dhall index b48c1dd..bf42326 100644 --- a/src/Interpreters/ParamsMember.dhall +++ b/src/Interpreters/ParamsMember.dhall @@ -166,23 +166,21 @@ let primitiveIsJsonb = let scalarIsJson = \(value : Model.Value) -> merge - { Primitive = primitiveIsJson, Custom = \(_ : Model.Name) -> False } + { Primitive = primitiveIsJson + , Custom = \(_ : Model.CustomTypeRef) -> False + } value.scalar let scalarIsJsonb = \(value : Model.Value) -> merge - { Primitive = primitiveIsJsonb, Custom = \(_ : Model.Name) -> False } + { Primitive = primitiveIsJsonb + , Custom = \(_ : Model.CustomTypeRef) -> False + } value.scalar let valueIsArray = - \(value : Model.Value) -> - Prelude.Optional.fold - Model.ArraySettings - value.arraySettings - Bool - (\(_ : Model.ArraySettings) -> True) - False + \(value : Model.Value) -> Prelude.Bool.not (Natural/isZero value.dimensionality) -- A bare (non-array) jsonb scalar binds via psycopg Jsonb(); a bare json scalar -- binds via Json(). json preserves the document verbatim, jsonb normalizes it, so @@ -253,10 +251,10 @@ let run = } in Prelude.Optional.fold - Model.Name + Model.CustomTypeRef value.scalar.customRef (Lude.Compiled.Type Output) - ( \(name : Model.Name) -> + ( \(ref : Model.CustomTypeRef) -> let dimsAtMostTwo = Natural/isZero (Natural/subtract 2 value.dims) @@ -289,7 +287,7 @@ let run = else Lude.Compiled.report Output [ input.pgName - , name.inSnakeCase + , ref.name.inSnakeCase ] "Array of an enum parameter with dimensionality > 2 is not supported" , Composite = @@ -314,15 +312,15 @@ let run = ) else Lude.Compiled.report Output - [ input.pgName, name.inSnakeCase ] + [ input.pgName, ref.name.inSnakeCase ] "Array of a composite type parameter with dimensionality > 1 is not supported" , Absent = Lude.Compiled.report Output - [ name.inSnakeCase ] + [ ref.name.inSnakeCase ] "Custom type not found in project customTypes" } - (lookup name) + (lookup ref.name) ) ( if isJsonArrayParam then Lude.Compiled.report diff --git a/src/Interpreters/Scalar.dhall b/src/Interpreters/Scalar.dhall index ba1e749..9fa4c3d 100644 --- a/src/Interpreters/Scalar.dhall +++ b/src/Interpreters/Scalar.dhall @@ -19,7 +19,7 @@ let ScalarDecode = < Passthrough | Custom > let Output = { pyType : Text , imports : ImportSet.Type - , customRef : Optional Model.Name + , customRef : Optional Model.CustomTypeRef , decode : ScalarDecode } @@ -35,18 +35,18 @@ let run = ( \(p : Primitive.Output) -> { pyType = p.pyType , imports = p.imports - , customRef = None Model.Name + , customRef = None Model.CustomTypeRef , decode = ScalarDecode.Passthrough } ) (Primitive.run {=} primitive) , Custom = - \(name : Model.Name) -> + \(ref : Model.CustomTypeRef) -> Lude.Compiled.ok Output - { pyType = name.inPascalCase + { pyType = ref.name.inPascalCase , imports = ImportSet.empty - , customRef = Some name + , customRef = Some ref , decode = ScalarDecode.Custom } } diff --git a/src/Interpreters/Value.dhall b/src/Interpreters/Value.dhall index f9717de..5ba1381 100644 --- a/src/Interpreters/Value.dhall +++ b/src/Interpreters/Value.dhall @@ -29,19 +29,21 @@ let run = Scalar.Output Output ( \(scalar : Scalar.Output) -> - Prelude.Optional.fold - Model.ArraySettings - input.arraySettings - Output - ( \(arraySettings : Model.ArraySettings) -> - let elementType = - if arraySettings.elementIsNullable + if Natural/isZero input.dimensionality + then { pyType = scalar.pyType + , imports = scalar.imports + , scalar + , dims = 0 + , elementIsNullable = False + } + else let elementType = + if input.elementIsNullable then "${scalar.pyType} | None" else scalar.pyType let arrayType = Natural/fold - arraySettings.dimensionality + input.dimensionality Text (\(inner : Text) -> "list[${inner}]") elementType @@ -49,16 +51,9 @@ let run = in { pyType = arrayType , imports = scalar.imports , scalar - , dims = arraySettings.dimensionality - , elementIsNullable = arraySettings.elementIsNullable + , dims = input.dimensionality + , elementIsNullable = input.elementIsNullable } - ) - { pyType = scalar.pyType - , imports = scalar.imports - , scalar - , dims = 0 - , elementIsNullable = False - } ) (Scalar.run {=} input.scalar) @@ -68,12 +63,12 @@ let qualifyCustom \(className : Text) -> \(value : Output) -> Prelude.Optional.fold - Model.Name + Model.CustomTypeRef value.scalar.customRef Text - ( \(name : Model.Name) -> + ( \(ref : Model.CustomTypeRef) -> Text/replace - name.inPascalCase + ref.name.inPascalCase (prefix ++ className) value.pyType ) From 46cbf79d7f15edc57cdaf9c4f45f4484b41a2922 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 20:02:10 +0300 Subject: [PATCH 05/14] Another plan --- ...-07-14-gen-contract-v5-sdk-v3-migration.md | 724 ++++++++++++++++++ 1 file changed, 724 insertions(+) create mode 100644 docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md diff --git a/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md b/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md new file mode 100644 index 0000000..4f48e60 --- /dev/null +++ b/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md @@ -0,0 +1,724 @@ +# Plan: bump gen-contract/gen-sdk pins and retire `buildLookup`'s `Text/equal` + +## Background + +This implements steps 3-4 of the "Next steps" section in +`docs/plans/2026-07-14-detext-equal-branch-filter.md` (item 4 of that +document's decision table). That document is the record of *why*; this +document is the *how*, broken into two tasks for subagent-driven execution. +Read the background doc if you want the full history — it is not required +reading for either task below, which are self-contained. + +Items 5-8 of the background doc (facade un-flattening, discarding collision +detection/rename mappings, dropping the ADR) are explicitly **out of scope** +for this plan — they need design work not yet done and are being handled +separately. + +## Global constraints + +- New pins (`dhall hash`-computed, already verified in the background doc): + ``` + -- src/Deps/Contract.dhall + https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall + sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef + + -- src/Deps/Sdk.dhall + https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall + sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 + ``` +- Verification oracle for both tasks: `mise run test` (drives real `pgn` + subprocesses against the local PostgreSQL server already running on + `localhost:5432`; `pgn` v0.9.1 is already installed via `mise`). Also run + `mise run lint`. Do **not** invent a separate `dhall` CLI invocation to + typecheck `.dhall` files standalone — this repo's `Text/equal` usage + (unrelated to this migration, e.g. in `PyIdent.dhall`, + `PythonNamespace.dhall`, `PythonNameMapping.dhall`, and the + `validateQueryMappings`/`validateCustomTypeMappings`/ + `validateCustomTypeIdentities` functions in `Project.dhall`) requires the + pgn-fork Dhall evaluator, which only `pgn generate` (invoked by the mise + tasks) provides locally. +- If `mise run test` fails because `pgn` v0.9.1 does not produce contract + data compatible with gen-contract v5.0.0 (e.g. it doesn't yet emit a + topologically-sorted `customTypes` list or the new `CustomTypeRef` shape), + stop and report `BLOCKED` with the exact failure — do not try to + work around a `pgn`-side incompatibility from the generator side. +- Do not touch decision-table items 1-3 (`emitSync`, custom-type + registration/`sameOrder`/`order` field, `PyIdent.dhall`'s reserved-name + disambiguation) — they are unrelated and already `Text/equal`-free. +- Do not touch `queryNameMappings`/`customTypeNameMappings` validation logic + or `PythonNameMapping.dhall` — their `Text/equal` usage is unrelated to + `buildLookup` and is scoped to items 6-7 (out of scope here). +- Regenerate the golden fixture (`mise run golden`) if `pgn generate` output + changes at all (it should not — this migration changes only the Dhall + generator's internals, not its emitted Python), and confirm + `tests/golden` has no diff. If it does diff, treat that as a signal + something changed unintentionally, not something to silently accept. +- Update the "Upcoming" section of `CHANGELOG.md` with one entry describing + the pin bump and the elimination of `buildLookup`'s `Text/equal` (after + Task 2). Follow the existing terse bullet style in that file. + +--- + +## Task 1: Bump pins to gen-contract v5.0.0 / gen-sdk v3.0.0 and fix compile breakage + +### Why this shape + +Bumping the pin is a breaking change independent of the `buildLookup` +migration (Task 2). Two things changed in the contract: + +1. `Scalar.Custom` changed from carrying a bare `Name` to carrying a + `CustomTypeRef`: + ```dhall + let CustomTypeRef = + { name : Name, pgSchema : Text, pgName : Text, index : Natural } + + let Scalar = < Primitive : Primitive | Custom : CustomTypeRef > + ``` + `index` is a 0-based position into `Project.customTypes`, and the + contract now *guarantees* that list is topologically sorted (every index + reachable from `customTypes[i]` is `< i`). `CustomTypeRef` still carries + `.name : Name`, so every site that only used the bare `Name` (e.g. to + feed the *existing* `Text/equal`-based lookup) can keep doing exactly + that by reading `.name` off the ref — this task does **not** change the + lookup mechanism, that is Task 2. + +2. `Value` flattened its optional array wrapper: + ```dhall + -- old (v4.0.1) + let ArraySettings = { dimensionality : Natural, elementIsNullable : Bool } + let Value = { arraySettings : Optional ArraySettings, scalar : Scalar } + + -- new (v5.0.0) + let Value = + { dimensionality : Natural + , elementIsNullable : Bool + , scalar : Scalar + } + ``` + `dimensionality = 0` now means "no array" (replacing + `arraySettings = None ArraySettings`); `elementIsNullable` is meaningless + when `dimensionality = 0`, same as before. There is no more + `Model.ArraySettings` type at all. + +This task's only job is: make every one of these sites compile again, +preserving current behavior exactly. Do not build the `List TypeKind` +lookup replacement here — that is Task 2, and mixing the two makes either +task hard to review independently. + +### Exact changes + +**`src/Deps/Contract.dhall`** — replace both lines with the new pin (URL and +hash both change): +``` +https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall + sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef +``` + +**`src/Deps/Sdk.dhall`** — replace both lines with the new pin: +``` +https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall + sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 +``` + +**`src/Interpreters/Scalar.dhall`** — `Output.customRef` changes type from +`Optional Model.Name` to `Optional Model.CustomTypeRef`, and the `Custom` +branch now receives a `CustomTypeRef` instead of a bare `Name`: +```dhall +let Output = + { pyType : Text + , imports : ImportSet.Type + , customRef : Optional Model.CustomTypeRef + , decode : ScalarDecode + } + +let run = + \(config : Config) -> + \(input : Input) -> + merge + { Primitive = + \(primitive : Model.Primitive) -> + Lude.Compiled.map + Primitive.Output + Output + ( \(p : Primitive.Output) -> + { pyType = p.pyType + , imports = p.imports + , customRef = None Model.CustomTypeRef + , decode = ScalarDecode.Passthrough + } + ) + (Primitive.run {=} primitive) + , Custom = + \(ref : Model.CustomTypeRef) -> + Lude.Compiled.ok + Output + { pyType = ref.name.inPascalCase + , imports = ImportSet.empty + , customRef = Some ref + , decode = ScalarDecode.Custom + } + } + input +``` +(Only the `customRef` field's type and the `Custom` branch's parameter +changed; `Primitive` branch changes only in the type annotation on +`None Model.CustomTypeRef`.) + +**`src/Interpreters/Value.dhall`** — `input.arraySettings` (an +`Optional Model.ArraySettings`) becomes two plain fields directly on +`Input`. Replace the `Prelude.Optional.fold` over `arraySettings` with a +check on whether `input.dimensionality` is zero: +```dhall +let run = + \(config : Config) -> + \(input : Input) -> + Lude.Compiled.map + Scalar.Output + Output + ( \(scalar : Scalar.Output) -> + if Natural/isZero input.dimensionality + then { pyType = scalar.pyType + , imports = scalar.imports + , scalar + , dims = 0 + , elementIsNullable = False + } + else let elementType = + if input.elementIsNullable + then "${scalar.pyType} | None" + else scalar.pyType + + let arrayType = + Natural/fold + input.dimensionality + Text + (\(inner : Text) -> "list[${inner}]") + elementType + + in { pyType = arrayType + , imports = scalar.imports + , scalar + , dims = input.dimensionality + , elementIsNullable = input.elementIsNullable + } + ) + (Scalar.run {=} input.scalar) +``` +`Output` (python.gen's own internal shape: `pyType`, `imports`, `scalar`, +`dims`, `elementIsNullable`) is unchanged — only how it's derived from +`Input` changes. `qualifyCustom` changes only its type annotation +(`Optional Model.Name` → `Optional Model.CustomTypeRef`) and reads +`.name.inPascalCase` off the ref instead of the bare name directly: +```dhall +let qualifyCustom + : Text -> Text -> Output -> Text + = \(prefix : Text) -> + \(className : Text) -> + \(value : Output) -> + Prelude.Optional.fold + Model.CustomTypeRef + value.scalar.customRef + Text + ( \(ref : Model.CustomTypeRef) -> + Text/replace + ref.name.inPascalCase + (prefix ++ className) + value.pyType + ) + value.pyType +``` + +**`src/Interpreters/Member.dhall`** — the `Custom` branch unwraps +`value.scalar.customRef` (now `Optional Model.CustomTypeRef`). Rename the +bound variable from `name` to `ref` for clarity and feed `ref.name` to +`lookup` (the lookup mechanism itself is untouched in this task — still +`Model.Name -> TypeKind`): +```dhall +, Custom = + Prelude.Optional.fold + Model.CustomTypeRef + value.scalar.customRef + (Lude.Compiled.Type Output) + ( \(ref : Model.CustomTypeRef) -> + let mkOutput = ... -- unchanged body, still uses `identity` + ... + in merge + { Enum = ... -- unchanged + , Composite = ... -- unchanged + , Absent = + Lude.Compiled.report + Output + [ ref.name.inSnakeCase ] + "Custom type not found in project customTypes" + } + (lookup ref.name) + ) + ( Lude.Compiled.report + Output + [ input.pgName ] + "Custom scalar without a customRef name" + ) +``` +Every other reference to the old `name : Model.Name` binding inside this +branch (the two `input.pgName, name.inSnakeCase` error-path lists) becomes +`input.pgName, ref.name.inSnakeCase`. Do not change anything about the +`Enum`/`Composite`/`Absent` merge arms themselves or the dims math. + +**`src/Interpreters/ParamsMember.dhall`** — identical shape of change as +Member.dhall, in the `Prelude.Optional.fold Model.Name value.scalar.customRef +...` block (around line 255-326): rename the bound `name : Model.Name` to +`ref : Model.CustomTypeRef`, change the fold's type argument to +`Model.CustomTypeRef`, call `lookup ref.name`, and update the two +`[input.pgName, name.inSnakeCase]` error-path lists to +`[input.pgName, ref.name.inSnakeCase]`, and the `Absent` arm's +`[name.inSnakeCase]` to `[ref.name.inSnakeCase]`. Also check +`isJsonbScalar`/`isJsonScalar`/`valueIsArray`/`scalarIsJson` earlier in this +file (used to compute `needsJsonbImport`/`needsJsonImport`/ +`isJsonArrayParam`) for any direct reads of `value.arraySettings` or +`Model.ArraySettings` — if present, adapt them the same way as Value.dhall +(`dimensionality`/`elementIsNullable` fields directly on `Model.Value` +instead of an `Optional Model.ArraySettings` wrapper). Read the full file +before editing; the excerpt in this brief only covered lines 200-339. + +**`src/Interpreters/CustomType.dhall`** — two things: +1. The `Composite` branch's member loop matches on `m.value.scalar` with + `Custom = \(name : Model.Name) -> ...` (checking + `m.value.arraySettings` to reject "Custom array fields inside a + composite type are not supported"). Update the merge arm's bound + variable type to `Model.CustomTypeRef` (rename `name` to `ref`, + references become `ref.name`/`ref.name.inSnakeCase`), and change the + `Optional Model.ArraySettings`/`m.value.arraySettings` check to + `Prelude.Bool.not (Natural/isZero m.value.dimensionality)` (i.e. "has at + least one array dimension") in place of testing whether the `Optional` + is `Some`. +2. `lookup input.name` (the self-consistency check, `input : Model.CustomType`) + is unaffected — `Model.CustomType.name` is still a plain `Model.Name`, + untouched by this contract version's breaking changes. Leave it as-is + in this task. + +**`src/Interpreters/Result.dhall`, `src/Interpreters/ResultColumns.dhall`, +`src/Interpreters/Query.dhall`** — these only carry `lookup : CustomKind.Lookup` +through as an opaque parameter (grep confirms no direct `Model.Name`/ +`Model.ArraySettings` access in these three files). They should not need any +edits; if `dhall`/`pgn generate` reports an error originating in one of +these files, that means this brief's premise was wrong somewhere upstream — +investigate rather than patching around it here. + +**`src/Interpreters/Project.dhall`** — do **not** touch `buildLookup`, +`effectiveResolvedCustomTypes`, or `CustomKind.Lookup` in this task (that's +Task 2). The only things in this file that might need attention from the +pin bump: grep the file yourself for any other `Model.Name`/ +`Model.ArraySettings` reads this brief didn't enumerate (the `Custom` merge +arm inside `resolveCustomTypes`'s `kind` computation, for instance, matches +on `customType.definition`, not `Scalar.Custom`, so it should be unaffected +— confirm this rather than assuming it). + +### Also check + +Run `grep -rn "Model.ArraySettings\|arraySettings" src/` yourself after +reading each file above — this brief's line numbers may drift slightly from +exact current file content. Fix every site the grep turns up, not just the +ones enumerated here. + +### Verification + +1. `mise run test` — must pass (73 passed, 0 skipped per the README's + documented baseline; note the exact count down if it differs and + investigate why before treating that as a pass). +2. `mise run lint`. +3. `mise run golden`, then `git diff tests/golden` — expect **no diff**. + If there is a diff, do not commit it silently; report it as a concern. +4. Report the exact commands run and their output in the task report. + +### Report file + +Write your report to the path given in the dispatch prompt. Include: which +files you touched beyond this brief's list (if any, and why), the full +`mise run test` summary line, and confirmation of a clean `git diff +tests/golden`. + +--- + +## Task 2: Replace `buildLookup`/`effectiveResolvedCustomTypes` with `Sdk.CustomTypes` + +**Depends on Task 1** (needs the bumped pins and `Model.CustomTypeRef` +plumbing already in place). Do not start this task until Task 1's review is +clean. + +### Why this shape + +`gen-sdk` v3.0.0 ships a ready-made, `Text/equal`-free replacement for +python.gen's own removal-cascade, in `Sdk.CustomTypes`: + +```dhall +-- List Bool: index i tells whether customTypes[i] is supported, given a +-- caller-supplied "is this kind of definition supported at all" predicate. +-- Composite/Domain members' own Custom refs are checked transitively via +-- List/fold over customTypes in order, since customTypes is topologically +-- sorted (every ref.index < the referencing type's own index). +supportedCustomTypes + : (Contract.CustomTypeDefinition -> Bool) -> + List Contract.CustomType -> List Bool + +-- Same fold, but returns Some instead of False +-- for each unsupported index (useful for warning messages). +supportedCustomTypesReasoned + : (Contract.CustomTypeDefinition -> Bool) -> + List Contract.CustomType -> List (Optional Contract.CustomTypeRef) + +customTypeIsSupported : List Bool -> Contract.CustomTypeRef -> Bool +queryIsSupported : List Bool -> Contract.Query -> Bool +``` + +This replaces the *removal-cascade* half of `buildLookup` + +`effectiveResolvedCustomTypes` (the repeated `Natural/fold` over +`Text/equal`-keyed lookups). It does **not** replace python.gen's own +rank-limit checks (array-of-enum >2 dims, array-of-composite >1 dim) — +those are python.gen-specific and stay, but they can now be expressed using +a referenced type's *kind* (Enum/Composite/Domain) obtained directly via +`ref.index`, instead of `Structures/CustomKind.dhall`'s +`Model.Name -> TypeKind` closure. + +**The subtlety that makes this not a mechanical swap:** the *kind* +classification of a custom type (is it an Enum, a Composite, or a Domain) +never changes across removal passes — it's a fixed fact about each type's +own definition. The *old* `buildLookup` conflated two separate concerns by +shrinking its entry list each pass: (a) "what kind is this referenced type" +and (b) "is this referenced type still a survivor" — a lookup miss meant +`Absent` regardless of which of the two failed, and `Absent` is what causes +a downstream compile to reject the reference (`"Custom type not found in +project customTypes"`). The new design must keep producing that same +`Absent`-on-either-failure behavior, or a composite that depends on a +removed type will wrongly keep compiling (it would still find the removed +type's *real* kind via a fixed classification, silently referencing a type +that no longer gets emitted). Concretely: build the kind classification +once (fixed, never shrinks), build the supported/removed cascade once (via +`Sdk.CustomTypes`), then mask the two together — a reference to an index +that is not "supported" reads as `Absent` regardless of its real kind, only +in `Skip` mode (in `Fail` mode nothing is masked, matching current +behavior exactly: `Fail` never drops anything, so every input is looked up +at its real kind and only fails via one `CustomTypeGen.run`/`Member`/ +`ParamsMember` error path, same as today). + +### Exact changes + +**`src/Structures/CustomKind.dhall`** — reshape `Lookup` from a closure to +an index-aligned list, and add an `at` accessor (mirrors gen-sdk's own +`optionalIndex` helper in `supportedCustomTypes.dhall`): +```dhall +let Model = ../Deps/Contract.dhall + +let Prelude = ../Deps/Prelude.dhall + +let Identity = + { className : Text, moduleName : Text, order : Natural } + +let TypeKind = + < Enum : Identity + | Composite : Identity + | Absent + > + +let Lookup = List TypeKind + +let at = + \(lookup : Lookup) -> + \(index : Natural) -> + Prelude.Optional.fold + TypeKind + (Prelude.List.index index TypeKind lookup) + TypeKind + (\(kind : TypeKind) -> kind) + TypeKind.Absent + +in { TypeKind, Lookup, Identity, at } +``` + +**Every call site that currently does `lookup ` where the lookup +was keyed by a `Model.Name`** switches to `CustomKind.at lookup ` +where `` comes from a `Model.CustomTypeRef.index` already in scope +(Task 1 already renamed the relevant bound variables to `ref`): + +- `src/Interpreters/Member.dhall`: `(lookup ref.name)` → `(CustomKind.at lookup ref.index)`. +- `src/Interpreters/ParamsMember.dhall`: `(lookup ref.name)` → `(CustomKind.at lookup ref.index)`. +- `src/Interpreters/CustomType.dhall`: + - The composite-member loop's `Custom = \(ref : Model.CustomTypeRef) -> ...` branch: + `(lookup ref.name)` → `(CustomKind.at lookup ref.index)`. + - The **self**-lookup (`lookup input.name`, appearing twice — once in the + `Enum` merge arm, once in the `Composite` merge arm of `run`) needs the + custom type's *own* index, which `Model.CustomType` does not carry + directly (unlike a *reference* to one). Add an explicit `index : Natural` + parameter to `CustomTypeGen.run`, threaded in by its caller + (`Interpreters/Project.dhall`, see below), and use + `CustomKind.at lookup index` in place of `lookup input.name` in both + merge arms: + ```dhall + let run = + \(config : Config) -> + \(lookup : CustomKind.Lookup) -> + \(index : Natural) -> + \(input : Input) -> + ... + in merge + { Enum = \(identity : CustomKind.Identity) -> ... + , Composite = \(_ : CustomKind.Identity) -> ... + , Absent = ... + } + (CustomKind.at lookup index) + ``` + (Both the `Enum`-definition branch's merge and the `Composite`-definition + branch's merge get this same treatment — each currently ends with + `(lookup input.name)`.) + +**`src/Interpreters/Project.dhall`** — this is where the real restructuring +happens. Replace `buildLookup`, `lookupEntries`'s dependents, and +`effectiveResolvedCustomTypes` as follows. Keep `IndexedCustomType`, +`ResolvedCustomType`, `LookupEntry`, `LookupKind`, `resolveCustomTypes`, and +`lookupEntries` exactly as they are today (they still correctly compute, +per-type, its Python identity and Enum/Composite/Domain kind tag, in +`input.customTypes` order — `resolveCustomTypes` already builds this via +`Prelude.List.indexed Model.CustomType customTypes`, so `entry.index` is +already the same 0-based position space as the contract's `ref.index`, +confirmed same order since `resolveCustomTypes` is called directly with +`input.customTypes`, never a reordered copy). + +Remove `buildLookup` entirely (lines ~590-611) and replace it with: + +```dhall +let kindOf + : List ResolvedCustomType -> CustomKind.Lookup + = \(entries : List ResolvedCustomType) -> + Prelude.List.map + ResolvedCustomType + CustomKind.TypeKind + ( \(rt : ResolvedCustomType) -> + merge + { Composite = CustomKind.TypeKind.Composite rt.lookupEntry.identity + , Enum = CustomKind.TypeKind.Enum rt.lookupEntry.identity + , Domain = CustomKind.TypeKind.Absent + } + rt.lookupEntry.kind + ) + entries + +let dimsAtMostTwo = + \(dims : Natural) -> Natural/isZero (Natural/subtract 2 dims) + +let dimsAtMostOne = + \(dims : Natural) -> Natural/isZero (Natural/subtract 1 dims) + +-- Own-definition support check fed to Sdk.CustomTypes.supportedCustomTypesReasoned. +-- `fixedKindOf` is the *unmasked* kind classification (kindOf above) — always +-- present regardless of survivorship, since kind never changes across passes; +-- only the array-rank check depends on it here. +let rankChecked + : CustomKind.Lookup -> Model.CustomTypeDefinition -> Bool + = \(fixedKindOf : CustomKind.Lookup) -> + \(definition : Model.CustomTypeDefinition) -> + merge + { Composite = + \(members : List Model.Member) -> + Prelude.List.all + Model.Member + ( \(member : Model.Member) -> + merge + { Primitive = \(_ : Model.Primitive) -> True + , Custom = + \(ref : Model.CustomTypeRef) -> + merge + { Enum = + \(_ : CustomKind.Identity) -> + dimsAtMostTwo member.value.dimensionality + , Composite = + \(_ : CustomKind.Identity) -> + dimsAtMostOne member.value.dimensionality + , Absent = True + } + (CustomKind.at fixedKindOf ref.index) + } + member.value.scalar + ) + members + , Enum = \(_ : List Model.EnumVariant) -> True + , Domain = \(_ : Model.Value) -> False + } + definition + +let boolAt = + \(bools : List Bool) -> + \(index : Natural) -> + Prelude.Optional.fold + Bool + (Prelude.List.index index Bool bools) + Bool + (\(b : Bool) -> b) + False +``` + +Then in `run`, replace the `effectiveResolvedCustomTypes`/ +`effectiveCustomTypes`/`lookup` block (currently lines ~1166-1204) with: + +```dhall + let resolvedCustomTypes = + resolveCustomTypes + resolvedConfig.customTypeNameMappings + input.customTypes + + let fixedKindOf = kindOf resolvedCustomTypes + + let supported + : List Bool + = if skip + then Prelude.List.map + (Optional Model.CustomTypeRef) + Bool + (Prelude.Optional.null Model.CustomTypeRef) + ( Sdk.CustomTypes.supportedCustomTypesReasoned + (rankChecked fixedKindOf) + input.customTypes + ) + else Prelude.List.map + ResolvedCustomType + Bool + (\(_ : ResolvedCustomType) -> True) + resolvedCustomTypes + + let lookup + : CustomKind.Lookup + = if skip + then Prelude.List.map + { index : Natural, value : CustomKind.TypeKind } + CustomKind.TypeKind + ( \(e : { index : Natural, value : CustomKind.TypeKind }) -> + if boolAt supported e.index then e.value else CustomKind.TypeKind.Absent + ) + (Prelude.List.indexed CustomKind.TypeKind fixedKindOf) + else fixedKindOf + + let indexedResolvedCustomTypes + : List { index : Natural, value : ResolvedCustomType } + = Prelude.List.indexed ResolvedCustomType resolvedCustomTypes + + let effectiveResolvedCustomTypes + : List { index : Natural, value : ResolvedCustomType } + = if skip + then Prelude.List.filter + { index : Natural, value : ResolvedCustomType } + (\(e : { index : Natural, value : ResolvedCustomType }) -> boolAt supported e.index) + indexedResolvedCustomTypes + else indexedResolvedCustomTypes + + let effectiveCustomTypes + : List { index : Natural, value : Model.CustomType } + = Prelude.List.map + { index : Natural, value : ResolvedCustomType } + { index : Natural, value : Model.CustomType } + ( \(e : { index : Natural, value : ResolvedCustomType }) -> + { index = e.index, value = e.value.value } + ) + effectiveResolvedCustomTypes +``` + +This changes `effectiveCustomTypes`'s element type from +`List Model.CustomType` to `List { index : Natural, value : Model.CustomType }` +everywhere it's consumed further down in `run`, so update: + +- `typesForCombine` — was + `Lude.Compiled.traverseList Model.CustomType CustomTypeGen.Output (\(ct : Model.CustomType) -> CustomTypeGen.run customTypeConfig lookup ct) effectiveCustomTypes` + becomes + ```dhall + Lude.Compiled.traverseList + { index : Natural, value : Model.CustomType } + CustomTypeGen.Output + ( \(e : { index : Natural, value : Model.CustomType }) -> + CustomTypeGen.run customTypeConfig lookup e.index e.value + ) + effectiveCustomTypes + ``` + (note the extra `e.index` argument, matching `CustomTypeGen.run`'s new + signature from this task's `CustomType.dhall` change above). +- Anywhere else `effectiveCustomTypes` is used as a plain + `List Model.CustomType` (check `combineOutputs`'s call and any other + reference in `run` — read the current file to find every use before + editing) needs either the same `{index,value}` shape or a + `Prelude.List.map ... (\(e) -> e.value) effectiveCustomTypes` to recover + the bare list, whichever the specific call site needs. `combineOutputs` + itself takes `customTypes : List CustomTypeGen.Output` (already-compiled + output, not `Model.CustomType`), which is unaffected — only the + *pre-compile* `effectiveCustomTypes`/`typesForCombine` wiring changes. + +**Remove entirely** (no longer needed, replaced by the above): +`typeSucceedsWith`, `typeWarningWith`'s use of a per-pass +`candidateLookup` parameter, and the `Natural/fold`-based +`effectiveResolvedCustomTypes` loop. If `skipWarnings` (further down in +`run`) still needs per-type warning `Report`s for `Skip` mode, rebuild it +using the *final* `lookup` (not a shrinking candidate) — i.e. +`typeWarningWith` becomes a plain function of `lookup` (no longer needing a +`candidateLookup` argument at all, since there is only one `lookup` now): +```dhall +let typeWarning = + \(ct : Model.CustomType) -> + \(index : Natural) -> + merge + { Ok = + \(_ : { value : CustomTypeGen.Output, warnings : List Report }) -> + None Report + , Err = + \(err : Report) -> Some { path = [ ct.name.inSnakeCase ] # err.path, message = err.message } + } + (CustomTypeGen.run customTypeConfig lookup index ct) +``` +and its use in `skipWarnings` maps over the **indexed** `input.customTypes` +(so each type is checked at its own real index, not filtered) — mirror +whatever indexing helper you already introduced above rather than +duplicating `Prelude.List.indexed` a third time. + +### Docs to update in this task + +- `README.md`: the paragraph starting "`fixtures/Exhaustive.dhall` is the + contract fixture. It and `buildLookup` rely on the pgn fork's `Text/equal` + builtin..." — `buildLookup` no longer exists or uses `Text/equal`. Correct + this paragraph to describe the current state: `fixtures/Exhaustive.dhall` + itself doesn't use `Text/equal`, but the generator's mapping-validation + code (`queryNameMappings`/`customTypeNameMappings`, + `validateCustomTypeIdentities`, `PyIdent.dhall`) still does, which is why + CI still needs the fork-aware evaluator. Don't claim `Text/equal` is gone + from the repo — it isn't (that's items 6-7, out of scope here). +- `DESIGN.md` section 10, "The pinned `Text/equal` constraint" — this + section specifically describes `buildLookup`'s mechanism, which this task + deletes. Rewrite it to describe the new `Sdk.CustomTypes`-based cascade + (or fold its content into a short note that `buildLookup` was replaced by + `gen-sdk`'s `CustomTypes` module and point at the remaining `Text/equal` + users listed above) — do not just delete the section silently; the + surrounding sections may reference it. +- `DESIGN.md` around line 278/289 (the `-> buildLookup(custom types)` pseudo + pipeline sketch, and "`onUnsupported: Skip` repeatedly rebuilds + `buildLookup`...") — update to describe the new one-pass + `Sdk.CustomTypes.supportedCustomTypesReasoned` mechanism instead of a + repeated rebuild. +- `CHANGELOG.md`: the existing "Retained `buildLookup`..." bullet (in a + past/released section, not "Upcoming") describes a past decision that no + longer holds. Do not edit released history; instead add a new bullet under + "Upcoming" (per Global Constraints) noting the v5.0.0/v3.0.0 bump and that + `buildLookup` was replaced by `gen-sdk`'s `CustomTypes` module. +- Leave `docs/upstream-asks.md` alone unless you find it makes a factual + claim about `buildLookup`'s *current* mechanism (as opposed to the + upstream ask itself, which is unrelated to this task) — read it first to + check. + +### Verification + +Same as Task 1: `mise run test` (expect the same passing count as Task 1 +left it at), `mise run lint`, `mise run golden` + `git diff tests/golden` +expecting no diff. Additionally: since this task touches the `Skip`-mode +cascade specifically, if this repo's test suite exercises +`onUnsupported: Skip` (check `tests/` for it), pay particular attention to +those tests passing — this is the behavior most at risk of a subtle +regression from this refactor (see the "Absent-on-either-failure" subtlety +above). + +### Report file + +Same contract as Task 1: report which files changed, the full `mise run +test` summary, confirmation of clean `git diff tests/golden`, and +explicitly confirm whether any `Skip`-mode-specific test exists and passed. From 76db5fadcfc761d1c5a18b3d917f6eeb46af38fe Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 13:33:11 +0300 Subject: [PATCH 06/14] python.gen: replace PyIdent's hand-rolled keyword equality with Lude.Text.replaceIfOneOf The prior double-Text/replace + digit-marker trick was justified by a claimed pgn Text/replace bug across concatenation boundaries that never actually reproduced (verified against plain dhall). Text/equal still can't be used (dropped in pgn 0.11.0, absent from the Dhall spec), so this swaps in the shared library's own sentinel-based equality helper instead, already used elsewhere in this codebase. Same public interface; golden output unchanged. --- src/Structures/PyIdent.dhall | 40 ++++++++---------------------------- 1 file changed, 9 insertions(+), 31 deletions(-) diff --git a/src/Structures/PyIdent.dhall b/src/Structures/PyIdent.dhall index c427d2d..e769a35 100644 --- a/src/Structures/PyIdent.dhall +++ b/src/Structures/PyIdent.dhall @@ -43,42 +43,20 @@ let pythonKeywords = , "yield" ] +let Lude = ../Deps/Lude.dhall + -- Add the caller's suffix when `name` collides with one of `reserved`. Callers -- keep the raw name for the SQL placeholder / dict key / row[...] lookup; only -- the Python identifier is sanitized. --- --- Equality without Text/equal: java.gen's escapeJavaKeyword delimiter trick does --- not apply here because pgn's embedded Text/replace misses needles spanning a --- text concatenation boundary (verified against the pinned pgn), so a "|"-wrapped --- name never matches its wrapped keyword. Bare replaces do work: --- `Text/replace name markTrue candidate` yields exactly `markTrue` only when --- `name` equals `candidate`; any mismatch residue keeps a letter of the --- alphabetic reserved word or grows past the digits-only marker, so it can never --- match inside `markTrue`, the second replace maps match to `name` and mismatch --- to `markTrue`, and the final replace rewrites `acc` on a match only. --- Limitations: a name containing the literal marker string is corrupted by the --- mismatch branch (the marker becomes the needle replaced in `acc`), and `acc` --- must be read exactly once per fold step or the expression re-embeds itself at --- every reserved word and blows up exponentially. -let markTrue = "0000000000000000000000000001" - -let suffixAgainst = - \(suffix : Text) -> +-- Dhall (and pgn's evaluator, as of pgn 0.11.0) has no Text/equal builtin, so +-- Lude.Text.replaceIfOneOf gets exact-match replacement via a sentinel-wrapped +-- Text/replace instead. +let suffixAgainst + : Text -> List Text -> Text -> Text + = \(suffix : Text) -> \(reserved : List Text) -> \(name : Text) -> - List/fold - Text - reserved - Text - ( \(candidate : Text) -> - \(acc : Text) -> - let signal = Text/replace name markTrue candidate - - let finalNeedle = Text/replace signal name markTrue - - in Text/replace finalNeedle (name ++ suffix) acc - ) - name + Lude.Text.replaceIfOneOf reserved (name ++ suffix) name let sanitizeAgainst = suffixAgainst "_" From 54d9ab7c9b58deab708ce6c88b3d0752507773dc Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 20:44:23 +0300 Subject: [PATCH 07/14] Bump pgn pin to v0.12.0 to unblock gen-contract v5 verification pgn v0.9.1 hard-rejects gen-contract major version 5 ("Incompatible contract major version: 5. Expected 4."), which blocked verifying Task 1's gen-contract v5.0.0 / gen-sdk v3.0.0 bump. pgn v0.12.0 adds gen-contract v5.0.0 support. --- mise.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mise.toml b/mise.toml index 00c0cfb..b185c3b 100644 --- a/mise.toml +++ b/mise.toml @@ -1,7 +1,7 @@ # This is a standalone repo (unlike the monorepo copy this was extracted from, # which inherits pgn from its repo-root mise.toml), so pgn is pinned here. [tools] -"github:pgenie-io/pgenie" = { version = "v0.9.1", exe = "pgn" } +"github:pgenie-io/pgenie" = { version = "v0.12.0", exe = "pgn" } uv = "latest" python = "3.12" From 0ce2836a91427c364167692366c305f8381c6bba Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Tue, 14 Jul 2026 20:51:21 +0300 Subject: [PATCH 08/14] Update mise.lock for pgn v0.12.0 Generated by mise install after the mise.toml pin bump. --- mise.lock | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mise.lock b/mise.lock index 2f2ab47..78f3772 100644 --- a/mise.lock +++ b/mise.lock @@ -1,18 +1,18 @@ # @generated - this file is auto-generated by `mise lock` https://mise.en.dev/dev-tools/mise-lock.html [[tools."github:pgenie-io/pgenie"]] -version = "0.9.1" +version = "0.12.0" backend = "github:pgenie-io/pgenie" [tools."github:pgenie-io/pgenie"."platforms.linux-x64"] -checksum = "sha256:6fd7d9852b43083e29e6b81311cad526773ee58fb34e3e2916c599a477828ae3" -url = "https://github.com/pgenie-io/pgenie/releases/download/v0.9.1/pgn-linux-x64.tar.gz" -url_api = "https://api.github.com/repos/pgenie-io/pgenie/releases/assets/472372948" +checksum = "sha256:27aab711ba18b76ce3daeb3e1f204ce404d0ad97c7952ad08b8f1c13c41703e3" +url = "https://github.com/pgenie-io/pgenie/releases/download/v0.12.0/pgn-linux-x64.tar.gz" +url_api = "https://api.github.com/repos/pgenie-io/pgenie/releases/assets/476896947" [tools."github:pgenie-io/pgenie"."platforms.macos-arm64"] -checksum = "sha256:17469d67bbb540af197df9b1abc30b097c4f222cb59bad53f8a3458929a0e5bd" -url = "https://github.com/pgenie-io/pgenie/releases/download/v0.9.1/pgn-macos-arm64.tar.gz" -url_api = "https://api.github.com/repos/pgenie-io/pgenie/releases/assets/472372949" +checksum = "sha256:9f95485d1607e3c1c3eebec9b4cde99163981c517d31b836b456228af0164206" +url = "https://github.com/pgenie-io/pgenie/releases/download/v0.12.0/pgn-macos-arm64.tar.gz" +url_api = "https://api.github.com/repos/pgenie-io/pgenie/releases/assets/476896949" [[tools.python]] version = "3.12.12" From 02fff4ffeed6701b8c52b966d83353f84f619bbe Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Wed, 15 Jul 2026 00:00:21 +0300 Subject: [PATCH 09/14] Discard Text/equal-dependent mapping/namespace validation pgn 0.11.0 (landed in Task 2) dropped the Text/equal builtin from its embedded Dhall evaluator, and there is no way to reconstruct a Bool or other branching decision from comparing two runtime Text values using only Text/replace. Every piece of this generator that made such a decision is removed: - queryNameMappings/customTypeNameMappings (Structures/PythonNameMapping.dhall and its two call sites in Query.dhall/CustomType.dhall) - the rename-mapping config feature, whole-cloth discarded per the plan's decision table. - validateCustomTypeIdentities (Interpreters/Project.dhall) - pushed upstream instead; pgenie-io/pgenie#75 asks pgn to guarantee unique custom-type identities at the source. - Structures/PythonNamespace.dhall's validate and its 4 call sites (per-query, per-custom-type, project-wide facade/module, and per-type module-internal namespace audits) - no longer expressible, but tests/test_generated.py's basedpyright --strict gate (errorCount == 0, warningCount == 0) already catches the same collisions one layer down, just later and less precisely attributed. Also removes PyIdent.dhall's isSnakeIdentifier/isPascalIdentifier helpers (dead code once the mapping-target validation they served is gone), the now-stale ADR 0001, and tests/test_python_name_collisions.py. Updates README/DESIGN/CHANGELOG to describe the new state honestly, and updates test_unsupported_types.py's synthetic contract-probe helper to the gen-contract v5.0.0 shape (CustomTypeRef, flattened dimensionality) so it typechecks again, independent of which task's changes it nominally covers. Repo-wide `grep -rn "Text/equal" src/` now turns up exactly one remaining functional use: Interpreters/Project.dhall's buildLookup (line 142) - that's the next task's scope (replacing it with gen-sdk's Sdk.CustomTypes). mise run test: 4 failed, 18 passed, 28 errors (up from 39 failed, 7 passed, 28 errors before this task) - every remaining failure traces to that same buildLookup Text/equal line, confirmed by inspecting each one individually. mise run golden also still fails at the same line (Exhaustive.dhall has custom types), so tests/golden is untouched - a null result, not a confirmation of no diff, pending the next task. --- CHANGELOG.md | 32 +- DESIGN.md | 78 +- README.md | 55 +- .../0001-generated-python-name-collisions.md | 74 -- ...-07-14-gen-contract-v5-sdk-v3-migration.md | 407 +++++++++- fixtures/Exhaustive.dhall | 4 - src/Interpreters/CustomType.dhall | 108 +-- src/Interpreters/Project.dhall | 724 +----------------- src/Interpreters/Query.dhall | 16 +- src/Structures/PyIdent.dhall | 62 -- src/Structures/PythonNameMapping.dhall | 48 -- src/Structures/PythonNamespace.dhall | 97 --- src/package.dhall | 9 +- tests/test_python_name_collisions.py | 692 ----------------- tests/test_unsupported_types.py | 51 +- 15 files changed, 491 insertions(+), 1966 deletions(-) delete mode 100644 docs/adr/0001-generated-python-name-collisions.md delete mode 100644 src/Structures/PythonNameMapping.dhall delete mode 100644 src/Structures/PythonNamespace.dhall delete mode 100644 tests/test_python_name_collisions.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 698d769..e9af5c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,13 +1,17 @@ # Upcoming -- Added fail-loud, namespace-wide validation for generated Python names. Query - and custom-type collisions can be resolved with typed, whole-entity mappings; - invalid, reserved, duplicate, and unknown mappings are rejected. Defensive - local-member audits identify both sources when such names reach the generator; - pgn may reject them earlier, and resolution is a SQL or schema rename. The - generator never silently overwrites files or assigns numeric suffixes. - Per-type module audits also reject a mapped class that would shadow an actual - primitive, core, or custom dependency import. +- Removed the `queryNameMappings`/`customTypeNameMappings` rename-mapping + config and all generation-time Python namespace-collision detection + (project-wide facade/module, per-query, per-custom-type, and per-type + module-internal audits). `pgn` 0.11.0 dropped the `Text/equal` builtin these + relied on to compare two runtime `Text` values, and there is no + `Text/replace`-based way to reconstruct that decision. A colliding schema is + no longer caught at `pgn generate`; it now fails the generated package's + `basedpyright --strict` gate instead (`reportRedeclaration` or + `reportInvalidTypeForm`), pointing at the generated Python rather than the + originating SQL/schema. + [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn + to guarantee unique custom-type identities at the source. - Finalized one additive package surface. Async functions remain at the package root for every configuration. `emitSync: true` adds `.sync`, a sync @@ -79,9 +83,10 @@ tuples with `dataclasses.fields` and `getattr`, preserving one-field arity. The fixture exercises the type as both a parameter and a result. -- Kept `PyIdent.dhall` independent of fork-only text equality by using its - bounded `Text/replace` marker construction. Keyword fields and parameters gain - a trailing underscore only in Python, while raw database names remain stable. +- Kept `PyIdent.dhall` independent of fork-only text equality by using + `Lude.Text.replaceIfOneOf`'s bounded, sentinel-wrapped `Text/replace` + construction. Keyword fields and parameters gain a trailing underscore only + in Python, while raw database names remain stable. - Preserved the release and license flow. The release job resolves `src/package.dhall` into the `resolved.dhall` release asset, then byte-checks @@ -95,3 +100,8 @@ carries a `CustomTypeRef` (name, pgSchema, pgName, index) instead of a bare `Name`, and `Value` flattens its optional array wrapper into plain `dimensionality`/`elementIsNullable` fields. + +- Bumped the pinned `pgn` tool to v0.12.0 (from v0.9.1), required for the + gen-contract v5.0.0 support above. `pgn` 0.11.0, a prerequisite, dropped the + `Text/equal`/`Text/length`/`Bool/equal` builtins from its embedded Dhall + evaluator to stay in line with the official Dhall spec. diff --git a/DESIGN.md b/DESIGN.md index a55e017..ab7e825 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -157,7 +157,7 @@ directly. Statement SQL is passed as `LiteralString`. No runtime accepts encoded SQL bytes. The generated package has no separately installed support library; its only non-stdlib consumer dependency is `psycopg>=3.3.4,<4`. -## 5. Configuration, mapping, and unsupported shapes +## 5. Configuration and unsupported shapes The public Dhall config is: @@ -165,14 +165,6 @@ The public Dhall config is: { packageName : Optional Text , emitSync : Optional Bool , onUnsupported : Optional < Fail | Skip > -, queryNameMappings : Optional - (List { source : Text, target : { snakeCase : Text, pascalCase : Text } }) -, customTypeNameMappings : Optional - ( List - { source : { schema : Text, name : Text } - , target : { snakeCase : Text, pascalCase : Text } - } - ) } ``` @@ -182,35 +174,25 @@ The public Dhall config is: - `packageName`: project name in kebab case; - `emitSync`: `False`; - `onUnsupported`: `Fail`. -- `queryNameMappings`: `[]`; -- `customTypeNameMappings`: `[]`. An omitted config block, an omitted field, and a `null` field therefore use the same fallback. Async output is present for every value of `emitSync`; only true adds sync output. -Source-derived identifiers first receive lexical and reserved-name escaping. -Typed mappings replace a complete query or custom-type Python identity and must -already contain exact valid targets. After resolution, project and local -namespace audits reject duplicate modules, functions, Row classes, custom -types, facade exports, parameters, fields, and enum members before rendering. -Each custom-type module also audits its class against the primitive, core, and -custom dependency symbols it actually imports. -Fixed core exports remain occupied. The generator never silently overwrites a -file or assigns an order-dependent numeric suffix; see -[ADR 0001](docs/adr/0001-generated-python-name-collisions.md). - -Custom-type mapping is exact only for entities preserved in the input contract. -pgn 0.9.1 can collapse same-unqualified-name types across schemas and expose an -unqualified `Scalar.Custom` reference. The generator cannot reconstruct the -discarded schema identity from SQL and rejects duplicate unqualified contract -names if they do reach it. Until upstream preserves all schema-qualified types -and a stable qualified reference, such database shapes are unsupported. - -Local member audits are defensive for inputs that reach the generator. pgn may -reject a conflicting SQL or schema spelling during analysis first. Local -conflicts are resolved by renaming that SQL or schema source; whole-entity query -and custom-type mappings do not apply to members. +Source-derived identifiers receive lexical and reserved-name escaping only +(`Structures/PyIdent.dhall`); there is no rename-mapping config and no +generation-time namespace-collision detection. `pgn`'s embedded Dhall evaluator +dropped `Text/equal` (pgn 0.11.0), which removed the only mechanism that could +compare two runtime `Text` values to decide a collision or resolve a mapping's +`source` against a query/type name; that decision is not reconstructible from +`Text/replace` alone (see section 10). Instead, the generated package is held +to `basedpyright --strict` returning zero errors and zero warnings; a +duplicate dataclass field name or a name that shadows a needed type surfaces +there (`reportRedeclaration`/`reportInvalidTypeForm`) rather than at `pgn +generate` time, and less precisely attributed (generated Python, not the +originating SQL/schema). +[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn +to guarantee unique custom-type identities at the source. `Interpreters/Primitive.dhall` maps pgn union constructors, not signature-file strings. Supported scalars are Boolean, integer and OID, floating point, @@ -348,10 +330,31 @@ fixed-point filtering, registration order, file selection, and facades. ## 10. The pinned `Text/equal` constraint +`pgn` 0.11.0 removed `Text/equal` (along with `Text/length` and `Bool/equal`) +from its embedded Dhall evaluator, to stay in line with the official Dhall +spec. There is no way to reconstruct a `Bool` or a differently-typed decision +from comparing two arbitrary runtime `Text` values using only `Text/replace`; +`Text/replace`-based tricks (this repository's own former keyword-marker +trick, and gen-sdk's `Lude.Text.replaceIfEqual`/`replaceIfOneOf`) only ever +transform text, they cannot branch into a different type. Every piece of this +generator that made a decision this way is gone: the `queryNameMappings`/ +`customTypeNameMappings` rename-mapping config +(`Structures/PythonNameMapping.dhall`, matched a mapping's `source` against a +runtime name), `validateCustomTypeIdentities` (rejected two custom types +collapsing to the same unqualified contract `Name`), and +`Structures/PythonNamespace.dhall`'s `validate` (4 call sites in +`Interpreters/Project.dhall`: per-query and per-custom-type local audits, the +project-wide facade/module audit, and per-type module-internal bindings). +[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn +to guarantee unique custom-type identities at the source instead. The +generated package's `basedpyright --strict` gate (see section 12) is now the +only backstop against a Python name collision reaching a consumer. + `buildLookup` intentionally remains in `Interpreters/Project.dhall`. It compares a custom reference's snake-case name with the project custom type name using `Text/equal`. That builtin belongs to pgn's embedded Dhall fork and is not -available in upstream standard Dhall. +available in upstream standard Dhall. It is, as of this writing, the only +remaining `Text/equal` use anywhere in `src/` — a repo-wide grep confirms it. The dependency is pinned and explicit. The complete fixture needs fork-aware evaluation solely because it invokes this generator and the local `buildLookup` @@ -362,9 +365,10 @@ text equality, that prevents pgn 0.9.1 from collapsing same-unqualified-name types across schemas. Until then, pgn and CI's pinned fork-aware action are the supported evaluators, and cross-schema duplicate type names are unsupported. -`PyIdent.dhall` uses its separate `Text/replace` marker construction for keyword -membership. `ImportSet.dhall` uses natural project indexes for equality, -deduplication, and ordering. Neither substitutes for the project lookup. +`PyIdent.dhall` uses `Lude.Text.replaceIfOneOf`'s bounded `Text/replace` +construction for keyword membership. `ImportSet.dhall` uses natural project +indexes for equality, deduplication, and ordering. Neither substitutes for the +project lookup. ## 11. Taking ownership diff --git a/README.md b/README.md index d06900e..6338461 100644 --- a/README.md +++ b/README.md @@ -43,8 +43,6 @@ freeze key machine-specific; pgn project configuration does not accept a | `packageName` | `Text` | project name in kebab case | | `emitSync` | `Bool` | `False` | | `onUnsupported` | `"Fail" \| "Skip"` | `"Fail"` | -| `queryNameMappings` | typed query-name mappings | `[]` | -| `customTypeNameMappings` | typed PostgreSQL type-name mappings | `[]` | The `config` block and every field are optional. An omitted or `null` field uses its default. Omitting `emitSync`, setting it to `null`, or setting it to `false` @@ -59,47 +57,18 @@ unsupported custom type, its dependent custom types and statements, and every affected facade, registration, and statement entry until the survivors are a strict-importable fixed point. -Generated Python namespaces are audited before files are rendered. Collisions -fail loudly instead of overwriting a file or receiving an order-dependent -numeric suffix. An intentional public rename uses one typed mapping for the -entity's snake-case module/function name and Pascal-case Row/type name: - -```yaml -queryNameMappings: - - source: sync_query - target: - snakeCase: synchronize - pascalCase: Synchronize -customTypeNameMappings: - - source: - schema: public - name: json_value - target: - snakeCase: pg_json_value - pascalCase: PgJsonValue -``` - -Mapping sources must identify an existing query or exact PostgreSQL schema/type -pair. A query source is pgn's snake-case query name; a custom-type source is the -exact PostgreSQL schema and type name preserved in the input contract. Targets -are final Python names: invalid or reserved targets are rejected, and -PostgreSQL names remain unchanged. -`PgJsonValue` above is a user-chosen Python alias for `public.json_value`, not a -built-in type or automatic naming rule. - -pgn 0.9.1 does not preserve two custom types that share one unqualified name -across schemas: it can retain only one `customTypes` entry, while -`Scalar.Custom` carries only an unqualified `Name`. A mapping cannot recover a -schema-qualified identity missing from the contract. Avoid that database shape -until upstream preserves every schema-qualified type and a stable qualified -reference; if a future input contains two entries with the same unqualified -contract `Name`, this generator rejects it as ambiguous. - -Local parameter, result-field, composite-field, and enum-member audits are -defensive for names that reach the generator. pgn may reject a conflicting SQL -or schema spelling earlier. Such a conflict is resolved at its SQL or schema -source; the entity mappings above do not apply. See -[ADR 0001](docs/adr/0001-generated-python-name-collisions.md). +Generated Python names are not audited for collisions at generation time. +Query and custom-type identifiers are derived directly from their PostgreSQL +source names (sanitized for Python syntax and reserved words only); a schema +that maps two different SQL entities onto the same Python identifier will not +be caught by `pgn generate`. Instead, the generated package is held to +`basedpyright --strict` with zero errors and zero warnings; a genuine +collision typically surfaces there as `reportRedeclaration` or +`reportInvalidTypeForm`, pointing at the generated Python rather than the +original SQL/schema source. Renaming the conflicting SQL or schema entity is +the fix. [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) +asks pgn to guarantee unique custom-type identities at the source, which would +let a future version of this generator catch such collisions earlier again. ## Generated package diff --git a/docs/adr/0001-generated-python-name-collisions.md b/docs/adr/0001-generated-python-name-collisions.md deleted file mode 100644 index ee8783b..0000000 --- a/docs/adr/0001-generated-python-name-collisions.md +++ /dev/null @@ -1,74 +0,0 @@ -# ADR 0001: Fail loudly on generated Python name collisions - -Status: Accepted - -## Context - -Different PostgreSQL names can resolve to the same Python name. For example, -the query names `sync` and `sync_query` both resolved to the module and function -`sync_query`, so one generated file silently replaced the other. Similar -collisions can affect Row classes, custom types, facade exports, and members -after Python keyword escaping. - -Silent overwrite loses code. Order-based suffixes such as `name2` also make a -public API change when an unrelated declaration is added or reordered. - -## Decision - -Lexical handling and uniqueness are separate stages: - -1. Source-derived defaults are escaped for Python keywords and generator-owned - implementation names. -2. Optional typed mappings resolve a declaration's complete Python identity. -3. Explicit targets must already be exact, valid, non-reserved Python names. - Snake-case targets start with a lowercase ASCII letter, and Pascal-case - targets start with an uppercase ASCII letter. Facade and module metadata - dunder names cannot be overwritten. Invalid targets are rejected rather - than silently rewritten. -4. The generator audits each Python namespace selected for emission after name - resolution and before rendering files. A collision stops generation and - identifies both owners and the final name. - -The greenfield configuration uses entity-typed mappings: - -```dhall -let PythonName = { snakeCase : Text, pascalCase : Text } - -let QueryNameMapping = { source : Text, target : PythonName } - -let CustomTypeNameMapping = - { source : { schema : Text, name : Text }, target : PythonName } -``` - -For a query, `snakeCase` owns the statement module and async/root public name; -the sync facade exports that same public name while its adjacent implementation -function adds `_sync`. `pascalCase` owns `${pascalCase}Row`. For a custom type, -`snakeCase` owns the type module and imports, while `pascalCase` owns the class, -facade export, and adapter registration. PostgreSQL and SQL names never change. -Duplicate or unknown mapping sources, invalid targets, and mapped collisions all -fail before files are emitted. Fixed package names such as `JsonValue`, -`NoRowError`, `register_types`, and `sync` cannot be remapped. `PgJsonValue` is -an example explicit user alias, not an automatic naming policy. - -Local parameter, result-field, composite-field, and enum-member audits are -defensive for inputs that reach the generator; pgn may reject the conflicting -source spelling earlier. They are resolved by renaming the SQL placeholder, SQL -alias, or schema member. A future local override, if needed, must use a typed, -parent-scoped mapping instead of a string path grammar. - -The structured custom-type source is exact only when the input contract -preserves the entity. pgn 0.9.1 can collapse types with the same unqualified -name across schemas, and `Scalar.Custom` has no qualified identity. A mapping -cannot recover that lost distinction. Such database shapes remain unsupported -until upstream preserves all qualified custom types and references. - -## Consequences - -- Generation cannot silently overwrite a file or export. -- Adding or reordering a declaration cannot renumber existing public names. -- Intentional renames are deterministic and propagate through all references. -- A conflicting source requires either a source-level rename or an explicit - typed mapping. -- Mapping configuration is more verbose than automatic suffixing, but it is - type-checked, and raw identifiers represented in the contract remain - unambiguous. diff --git a/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md b/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md index 4f48e60..b96e2c2 100644 --- a/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md +++ b/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md @@ -5,14 +5,23 @@ This implements steps 3-4 of the "Next steps" section in `docs/plans/2026-07-14-detext-equal-branch-filter.md` (item 4 of that document's decision table). That document is the record of *why*; this -document is the *how*, broken into two tasks for subagent-driven execution. +document is the *how*, broken into tasks for subagent-driven execution. Read the background doc if you want the full history — it is not required -reading for either task below, which are self-contained. - -Items 5-8 of the background doc (facade un-flattening, discarding collision -detection/rename mappings, dropping the ADR) are explicitly **out of scope** -for this plan — they need design work not yet done and are being handled -separately. +reading for any task below, which are self-contained. + +**Scope grew mid-execution.** Task 1 (bumping gen-contract/gen-sdk pins) +could not be verified against `pgn` v0.9.1 — it rejects gen-contract major +version 5 outright. `pgn` v0.12.0 fixes that, but v0.11.0 (a prerequisite) +dropped the `Text/equal` builtin from its embedded Dhall evaluator entirely. +That forced a live investigation (documented in Task 2's "Why this exists" +and Task 3's "Why this shape") which found that `Text/equal`-free rework is +*impossible* (not just hard) for anything that makes a decision from +comparing two arbitrary runtime strings — which pulled decision-table items +6-7 (same-kind collision detection, rename mappings) and the ADR into scope +now, plus `PythonNamespace.dhall`'s collision detection more broadly (never +part of items 6-7). Task 3 covers all of that. Item 5 (facade +un-flattening) remains **out of scope** — it's unrelated to `Text/equal` and +still needs its own design pass. ## Global constraints @@ -26,28 +35,23 @@ separately. https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 ``` -- Verification oracle for both tasks: `mise run test` (drives real `pgn` +- Verification oracle for every task: `mise run test` (drives real `pgn` subprocesses against the local PostgreSQL server already running on - `localhost:5432`; `pgn` v0.9.1 is already installed via `mise`). Also run - `mise run lint`. Do **not** invent a separate `dhall` CLI invocation to - typecheck `.dhall` files standalone — this repo's `Text/equal` usage - (unrelated to this migration, e.g. in `PyIdent.dhall`, - `PythonNamespace.dhall`, `PythonNameMapping.dhall`, and the - `validateQueryMappings`/`validateCustomTypeMappings`/ - `validateCustomTypeIdentities` functions in `Project.dhall`) requires the - pgn-fork Dhall evaluator, which only `pgn generate` (invoked by the mise - tasks) provides locally. -- If `mise run test` fails because `pgn` v0.9.1 does not produce contract - data compatible with gen-contract v5.0.0 (e.g. it doesn't yet emit a - topologically-sorted `customTypes` list or the new `CustomTypeRef` shape), - stop and report `BLOCKED` with the exact failure — do not try to - work around a `pgn`-side incompatibility from the generator side. + `localhost:5432`). Also run `mise run lint`. Do **not** invent a separate + `dhall` CLI invocation to typecheck `.dhall` files standalone — the + pgn-fork Dhall evaluator (needed for anything using `Text/equal`, before + Task 3 removes the remaining uses) is only available locally through + `pgn generate` (invoked by the mise tasks). +- Task 1 was written and dispatched before Task 2/3 existed, when `pgn` + v0.9.1 was still pinned — its own text still says "unrelated `Text/equal` + usage" and "out of scope"; that was correct *at the time* for Task 1's + own diff, but Tasks 2-3 below supersede those scoping notes for the + branch as a whole. Follow each task's own instructions as written. - Do not touch decision-table items 1-3 (`emitSync`, custom-type registration/`sameOrder`/`order` field, `PyIdent.dhall`'s reserved-name - disambiguation) — they are unrelated and already `Text/equal`-free. -- Do not touch `queryNameMappings`/`customTypeNameMappings` validation logic - or `PythonNameMapping.dhall` — their `Text/equal` usage is unrelated to - `buildLookup` and is scoped to items 6-7 (out of scope here). + disambiguation) — they are unrelated and already `Text/equal`-free (aside + from the `py-ident-fix` cleanup pulled in by Task 2, which keeps the same + behavior). - Regenerate the golden fixture (`mise run golden`) if `pgn generate` output changes at all (it should not — this migration changes only the Dhall generator's internals, not its emitted Python), and confirm @@ -55,7 +59,7 @@ separately. something changed unintentionally, not something to silently accept. - Update the "Upcoming" section of `CHANGELOG.md` with one entry describing the pin bump and the elimination of `buildLookup`'s `Text/equal` (after - Task 2). Follow the existing terse bullet style in that file. + Task 4). Follow the existing terse bullet style in that file. --- @@ -64,7 +68,7 @@ separately. ### Why this shape Bumping the pin is a breaking change independent of the `buildLookup` -migration (Task 2). Two things changed in the contract: +migration (Task 4). Two things changed in the contract: 1. `Scalar.Custom` changed from carrying a bare `Name` to carrying a `CustomTypeRef`: @@ -80,7 +84,7 @@ migration (Task 2). Two things changed in the contract: `.name : Name`, so every site that only used the bare `Name` (e.g. to feed the *existing* `Text/equal`-based lookup) can keep doing exactly that by reading `.name` off the ref — this task does **not** change the - lookup mechanism, that is Task 2. + lookup mechanism, that is Task 4. 2. `Value` flattened its optional array wrapper: ```dhall @@ -102,7 +106,7 @@ migration (Task 2). Two things changed in the contract: This task's only job is: make every one of these sites compile again, preserving current behavior exactly. Do not build the `List TypeKind` -lookup replacement here — that is Task 2, and mixing the two makes either +lookup replacement here — that is Task 4, and mixing the two makes either task hard to review independently. ### Exact changes @@ -306,7 +310,7 @@ investigate rather than patching around it here. **`src/Interpreters/Project.dhall`** — do **not** touch `buildLookup`, `effectiveResolvedCustomTypes`, or `CustomKind.Lookup` in this task (that's -Task 2). The only things in this file that might need attention from the +Task 4). The only things in this file that might need attention from the pin bump: grep the file yourself for any other `Model.Name`/ `Model.ArraySettings` reads this brief didn't enumerate (the `Custom` merge arm inside `resolveCustomTypes`'s `kind` computation, for instance, matches @@ -339,11 +343,316 @@ tests/golden`. --- -## Task 2: Replace `buildLookup`/`effectiveResolvedCustomTypes` with `Sdk.CustomTypes` +## Task 2: Bump pgn to v0.12.0, pull in the `py-ident-fix` cleanup, re-verify Task 1 + +### Why this exists + +Task 1 landed correctly (commit `d18398c`) but could not be verified: +`pgn` v0.9.1 (pinned in `mise.toml`) hard-rejects gen-contract major version 5 +("Incompatible contract major version: 5. Expected 4."). `pgn` v0.12.0 +(released 2026-07-14) adds gen-contract v5.0.0 support — confirmed via its +changelog: *"Bump the `gen-contract` dependency to v5.0.0... Add +`CustomTypeRef`... Flatten `Value`'s `arraySettings`..."*, exactly matching +Task 1's changes. + +`pgn` v0.11.0 (a prerequisite of v0.12.0) is a breaking change: +*"Update to Dhall that lacks `Text/equal` to stay in line with the official +Dhall spec."* Confirmed empirically (both against the real `pgn` fork via a +throwaway `pgn generate` run, and against stock upstream `dhall` 1.42.3): +`Text/equal`, `Text/length`, and `Bool/equal` are all unavailable — there is +no way to produce a `Bool` or a differently-typed decision (e.g. an `Ok`/`Err` +union) from comparing two arbitrary runtime `Text` values using only +`Text/replace`. `Text/replace`-based tricks (this repo's old digit-marker +trick, and gen-sdk's own `Lude.Text.replaceIfEqual`/`replaceIfOneOf`) only +ever produce more `Text` — they can transform text, not decide anything. This +is why Task 3 (below) has to discard rather than "rework" several checks. + +Branch `py-ident-fix` (already pushed, commit `7685c0a8794b0d4e20947c2ff6aa9c11b4dabd48`, +"python.gen: replace PyIdent's hand-rolled keyword equality with +Lude.Text.replaceIfOneOf") already did the one part of this that *is* just a +transform (not a decision): `PyIdent.dhall`'s Python-keyword-suffixing. Pull +that commit's change into this branch as part of this task — it is a strict +simplification of code Task 1 already touches, with the same public +interface and unchanged golden output per its own commit message. + +### Exact changes + +1. **Cherry-pick or manually reapply** commit `7685c0a8794b0d4e20947c2ff6aa9c11b4dabd48` + from `py-ident-fix` onto this branch (`git log py-ident-fix` to find it; + `git cherry-pick 7685c0a8794b0d4e20947c2ff6aa9c11b4dabd48` is the + straightforward route — resolve any conflict against Task 1's changes by + keeping Task 1's contract-type changes and `py-ident-fix`'s + `sanitizeAgainst`/`Lude.Text.replaceIfOneOf` rewrite; they don't overlap + in `PyIdent.dhall`, which Task 1 never touched). +2. **`mise.toml`**: change the pgn tool pin from `v0.9.1` to `v0.12.0`: + ```toml + "github:pgenie-io/pgenie" = { version = "v0.12.0", exe = "pgn" } + ``` + Run `mise install` (or just let the next `mise run` task trigger it) to + fetch the new binary. +3. Re-run Task 1's verification now that the environment is unblocked: + `mise run test`, `mise run lint`, `mise run golden` + `git diff + tests/golden` (expect no diff). Do not make further Dhall changes in this + task beyond the cherry-pick above — if `mise run test` still fails for a + reason unrelated to the pgn version (e.g. a genuine bug in Task 1's + compile fixes), report it as a concern rather than silently patching + Task 1's work under this task's commit. + +### Verification + +`mise run test`, `mise run lint`, `mise run golden` + `git diff +tests/golden` (expect no diff). Report the exact `mise run test` summary +(pass/fail counts) — this is the first time it can actually run end to end +since Task 1 started, so capture it precisely. + +### Report file + +`.superpowers/sdd/task-2-report.md`. State clearly: the cherry-picked commit +hash (or, if reapplied manually, why), the new `mise run test` result, and +confirmation of clean `git diff tests/golden`. + +--- + +## Task 3: Discard `Text/equal`-dependent validation (mappings, identity check, namespace collision detection) + +**Depends on Task 2** (needs `pgn` v0.12.0 actually working, so this task's +own verification is meaningful). Do not start until Task 2's review is clean. + +### Why this shape + +With `Text/equal` gone from `pgn`'s embedded Dhall evaluator (confirmed in +Task 2's background), every piece of this generator that makes a *decision* +by comparing two arbitrary runtime `Text` values is no longer expressible — +not "hard to rework," genuinely impossible in the target language (see +Task 2's background section for the proof). Three clusters of code do this: + +1. **`queryNameMappings`/`customTypeNameMappings`** (the rename-mapping + config feature): `PythonNameMapping.dhall`'s `resolveQuery`/ + `resolveCustomType` match a mapping's `source` against a query/type name + via `Text/equal`. This is decision-table item 7 from the background doc + (`docs/plans/2026-07-14-detext-equal-branch-filter.md`) — already marked + **discard entirely**, independent of this new finding. +2. **`validateCustomTypeIdentities`** (`Interpreters/Project.dhall`, checks + whether two different custom types collapse to the same unqualified + contract `Name`): matches decision-table item 6's philosophy exactly — + "pushed upstream instead," and + [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) + (already filed) explicitly asks pgn to "guarantee unique custom-type + identities at the source." Discard, same rationale as item 6. +3. **`PythonNamespace.dhall`'s `validate`** (4 call sites in + `Interpreters/Project.dhall`: per-query local param/result-field + namespace, per-custom-type local field namespace, the project-wide + facade/module namespace, and per-type module-internal bindings): this + goes beyond items 6-7 (the local audits were never proposed for + discard), but it's equally impossible now. Confirmed there is a working + safety net one layer down: `tests/test_generated.py`'s + `test_generated_passes_basedpyright_strict` (and the same pattern in + `test_identifier_collisions.py`, `test_psycopg_adapter_contract.py`, + `test_unsupported_types.py`) already asserts + `errorCount == 0 and warningCount == 0` on the generated package. Verified + empirically: a duplicate dataclass field name triggers basedpyright's + `reportRedeclaration` (warning — fails the gate), and a field name that + shadows a type needed elsewhere triggers `reportInvalidTypeForm` (hard + error). The tradeoff is real and accepted deliberately: today a collision + fails fast at `pgn generate` with a message pointing at the SQL/schema + source; after this task, the same collision still fails the test suite, + just later (at basedpyright time) and less precisely attributed + (pointing at generated Python, not SQL). This was an explicit, discussed + tradeoff, not an oversight — do not try to preserve the old error + messages by some other means. + +### Exact changes + +**Delete entirely:** +- `src/Structures/PythonNameMapping.dhall` +- `src/Structures/PythonNamespace.dhall` +- `docs/adr/0001-generated-python-name-collisions.md` (its entire content + describes the mechanism being removed in this task — stale documentation + actively describing a removed feature is worse than no doc at all; do not + wait for item 8's broader doc pass) +- `tests/test_python_name_collisions.py` (this file, per the background + doc, "specifically tests the collision-detection and rename-mapping + mechanisms being discarded") + +**`src/package.dhall`**: remove `queryNameMappings`/`customTypeNameMappings` +from `Config` and `Config/default`, and the `PythonNameMapping` import. + +**`src/Interpreters/Project.dhall`**: remove +- `queryNameMappings`/`customTypeNameMappings` from `Config` and + `ResolvedConfig`, and their resolution in `run`. +- `validateCustomTypeIdentities`, `validateQueryMappings`, + `validateCustomTypeMappings`, `finishMappingValidation` (if nothing else + uses it after the other three are gone — check), `QueryMappingState`, + `CustomTypeMappingState`, `CustomTypeIdentity`, `CustomTypeIdentityState`, + and the `mappingsValid`/`mappingsAndLocalsValid` wiring in `run` that + chains them together (fold their removal into whatever the remaining + validation chain becomes — read the full `run` function first to see + what `mappingsAndLocalsValid`/`combined` looks like once + `validateLocalNamespaces` is *also* gone, see next point, since both + disappear in the same task). +- `validateLocalNamespaces`, `validateProjectNamespaces`, and the + `moduleValidation` block inside it — all of `PythonNamespace.validate`'s + 4 call sites. Remove the `PythonNamespace` import. +- The `PythonNameMapping` import, and any leftover reference to the + mapping/namespace types in this file's other functions (grep after + editing to confirm nothing dangles). + +**`src/Interpreters/Query.dhall`** and **`src/Interpreters/CustomType.dhall`**: +remove `queryNameMappings`/`customTypeNameMappings` from their `Config` +types, remove the `PythonNameMapping` import, and replace the +`PythonNameMapping.resolveQuery`/`resolveCustomType` call with the bare +default name directly — e.g. in `CustomType.dhall`: +```dhall +-- before +let pythonName = + PythonNameMapping.resolveCustomType + config.customTypeNameMappings + { schema = input.pgSchema, name = input.pgName } + { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase + , pascalCase = PyIdent.pySafeName input.name.inPascalCase + } + +-- after +let pythonName = + { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase + , pascalCase = PyIdent.pySafeName input.name.inPascalCase + } +``` +and the analogous simplification in `Query.dhall` around its +`PythonNameMapping.resolveQuery` call. Read both call sites in full before +editing — this brief doesn't reproduce their exact surrounding code. + +**`CustomType.dhall`'s `moduleBindings`/namespace-binding plumbing** +(`enumNamespaceBindings`, `compositeNamespaceBindings`, `moduleNamespace`, +`moduleBinding`, and the `Output.moduleBindings` field): these exist solely +to feed `PythonNamespace.validate`'s per-module check (now gone). Remove +them and the `moduleBindings` field from `CustomType.dhall`'s `Output`, and +its consumption in `Project.dhall`. Confirm nothing else reads +`moduleBindings` before deleting it (grep first). + +**Also check** `Interpreters/ResultColumns.dhall`, `Result.dhall`, +`Member.dhall`, `ParamsMember.dhall` for any `PythonNamespace`/ +`PythonNameMapping` references this brief didn't enumerate (expected: none, +but confirm). + +### Tests + +- Delete `tests/test_python_name_collisions.py` (see above). +- Read `tests/test_identifier_collisions.py` in full before touching it: it + covers query-name-vs-implementation-global collisions (decision-table + item 3's cluster, `PyIdent.dhall`'s reserved-name disambiguation — kept, + not part of this task) and *also* uses basedpyright directly (lines + ~282-290). Update only the parts of it that exercise the + now-deleted `queryNameMappings`/`customTypeNameMappings`/ + `PythonNamespace`-based paths, if any — keep everything about reserved + implementation-name disambiguation as-is. +- Read `tests/test_unsupported_types.py` in full — it references + `PythonNameMapping`/mapping config (per the earlier grep). Update it to + drop mapping-specific scenarios while keeping its `onUnsupported: Skip` + coverage intact (that coverage matters for Task 4, next). +- **`tests/test_unsupported_types.py`'s `_write_contract_probe` helper** + (around line 115, feeding the 6-case parametrized + `test_custom_shape_contracts_fail_loudly`) hand-authors a synthetic Dhall + wrapper module as a Python string, and needs updating for *this* task's + changes specifically (Task 4 will need a second pass on the same helper + for its own changes — see Task 4's Tests section below, don't try to + anticipate that part here): + - `interpreter_config` currently emits + `{ customTypeNameMappings = [] : List PythonNameMapping.CustomType }` + when `interpreter == "CustomType"` — `PythonNameMapping` no longer + exists after this task. Since `CustomType.dhall`'s `Config` also drops + `customTypeNameMappings` in this task (per the `Interpreters/*.dhall` + changes above), the interpreter config becomes `{=}` unconditionally + (same as the non-`CustomType` branch) — the `if interpreter == + "CustomType" else` branch can go entirely. + - The wrapper's own `let PythonNameMapping = ./Structures/PythonNameMapping.dhall` + import line must be removed. + - This probe is still on the *old* contract shape from before Task 1 (it + was never updated when Task 1 landed, since Task 1 didn't know test + fixtures existed at this level — confirmed by code review): its + `member.value` still builds `{ arraySettings = {array_settings}, scalar + = Model.Scalar.Custom name }` where `array_settings` renders + `"None Model.ArraySettings"` or `"Some { dimensionality = N, + elementIsNullable = False }"`, and `Model.Scalar.Custom name` passes a + bare `Model.Name`. Both no longer typecheck against the now-bumped + gen-contract v5.0.0 pin (Task 1). Fix this probe to the new shape as + part of *this* task (it's blocking verification regardless of which + task's changes it's nominally testing): `value` becomes + `{ dimensionality = {dimensionality}, elementIsNullable = False, scalar + = Model.Scalar.Custom { name, pgSchema = "public", pgName = + "probe_value", index = 0 } }` (drop the `array_settings` local + entirely, inline `dimensionality` directly — `0` for the non-array + case, same as today's `None` case). Confirm the probe still produces + the same 6 pass/fail outcomes the test asserts. + - Leave the probe's `CustomKind.Lookup` construction + (`\(_ : Model.Name) -> {lookup}`) as-is in this task — that's Task 4's + job, since `CustomKind.Lookup`'s shape doesn't change until then. +- Do **not** touch `tests/test_takeover_contract.py` — it's about the flat + facade (item 5), out of scope for this whole plan. +- Run the full suite; if new failures appear that trace to bindings this + task didn't anticipate removing, investigate rather than papering over + them (e.g. with a stub mapping). + +### Docs + +- **`README.md`**: remove the `queryNameMappings`/`customTypeNameMappings` + rows from the config table (lines ~46-47 as of this writing); remove the + "Generated Python namespaces are audited..." paragraph and its YAML + mapping example (~lines 62-96, read current content — line numbers will + have drifted from Task 1/2's edits); remove the ADR 0001 link. Replace + with a short, honest paragraph: namespace collisions are not detected at + generation time; a colliding schema fails `basedpyright --strict` on the + generated package instead (link to + [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) for + the upstream ask to prevent custom-type identity collisions at the + source). Don't overclaim — collisions are *possible*, just caught later. +- **`DESIGN.md`** section 5 ("Configuration, mapping, and unsupported + shapes"): remove the mapping-specific config fields/description and the + ADR reference; keep the `onUnsupported` description (unrelated). Section + 10 ("The pinned `Text/equal` constraint"): this section is also rewritten + by Task 4 for `buildLookup` specifically — in *this* task, update it (or + leave a marker for Task 4) to note that mapping/namespace/identity + validation no longer uses `Text/equal` either, and enumerate what (if + anything) still does after this task (check with a repo-wide grep for + `Text/equal` once done — there should be very little, if anything, left + outside `Interpreters/Project.dhall`'s `buildLookup`, which Task 4 + removes next). +- **`CHANGELOG.md`**: add an "Upcoming" bullet (matching the file's terse + style) noting the removal of `queryNameMappings`/`customTypeNameMappings` + and generation-time namespace-collision detection, with the basedpyright + rationale, and a bullet for the pgn v0.12.0 bump (Task 2, if not already + added there). Also fix the now-stale released-history bullet "Kept + `PyIdent.dhall` independent of fork-only text equality by using its + bounded `Text/replace` marker construction" (~line 82 as of this + writing) — Task 2's cherry-pick already replaced that exact mechanism + with `Lude.Text.replaceIfOneOf`. Don't rewrite released history in place; + either amend that bullet's wording to describe the current mechanism + accurately (it's still describing *current* behavior, just the wrong + implementation detail) or add a one-line "Upcoming" bullet superseding it + — whichever fits this file's existing convention for describing a + changed implementation detail of already-released behavior. + +### Verification + +`mise run test`, `mise run lint`, `mise run golden` + `git diff +tests/golden` (a diff here IS expected in this task, since removing +mapping config could change nothing about default-path output, but confirm +this — if `tests/golden` changes, explain exactly why in the report). + +### Report file + +`.superpowers/sdd/task-3-report.md`. Include: full file list touched, the +`mise run test` summary, confirmation of `tests/golden` status (diff or no +diff, with explanation either way), and a repo-wide `grep -rn "Text/equal" +src/` output so the next task (Task 4) knows exactly what's left. + +--- + +## Task 4: Replace `buildLookup`/`effectiveResolvedCustomTypes` with `Sdk.CustomTypes` -**Depends on Task 1** (needs the bumped pins and `Model.CustomTypeRef` -plumbing already in place). Do not start this task until Task 1's review is -clean. +**Depends on Tasks 1-3** (needs the bumped pins, working `pgn` v0.12.0, and +`Model.CustomTypeRef` plumbing already in place). Do not start this task +until Task 3's review is clean. ### Why this shape @@ -706,6 +1015,34 @@ duplicating `Prelude.List.indexed` a third time. upstream ask itself, which is unrelated to this task) — read it first to check. +### Tests + +`tests/test_unsupported_types.py`'s `_write_contract_probe` helper (around +line 115, feeding the parametrized `test_custom_shape_contracts_fail_loudly`) +hand-authors a synthetic Dhall wrapper module as a Python string. Task 3 +already updated it for the v5 contract shape and removed its +`PythonNameMapping` reference; this task needs one more pass, since +`CustomKind.Lookup` changes shape here: +- Its `let lookup : CustomKind.Lookup = \(_ : Model.Name) -> {lookup}` + construction (a `Model.Name -> TypeKind` closure) must become a + `List CustomKind.TypeKind` — for these probes there's exactly one custom + type in play ("ProbeValue"), so `lookup` becomes a one-element list: + `let lookup : CustomKind.Lookup = [ {lookup} ]`. +- `CustomType.dhall`'s `run` gained an explicit `index : Natural` parameter + in this task (the self-lookup consistency check). The probe's call site + (`Target.run interpreterConfig lookup {target_input}`) needs an index + argument threaded in when `interpreter == "CustomType"` — `0` (matching + the single-element `lookup` list above). For the `Member`-interpreter + probes (`interpreter == "Member"`), `Member.dhall`'s `run` signature is + unaffected by this task (it never needed a self-index, only the + already-present `ref.index` off the value being looked up) — don't add + an index argument there. +- Confirm all cases of `test_custom_shape_contracts_fail_loudly` still + produce the same pass/fail outcomes and error messages after this + change — this is the test most likely to catch a subtle regression in + the new `Absent`-on-either-failure masking logic (see this task's "Why + this shape" section above). + ### Verification Same as Task 1: `mise run test` (expect the same passing count as Task 1 diff --git a/fixtures/Exhaustive.dhall b/fixtures/Exhaustive.dhall index 6bb6fb9..5b5f2e6 100644 --- a/fixtures/Exhaustive.dhall +++ b/fixtures/Exhaustive.dhall @@ -18,8 +18,6 @@ let Gen = ../src/package.dhall let OnUnsupported = ../src/Structures/OnUnsupported.dhall -let PythonNameMapping = ../src/Structures/PythonNameMapping.dhall - let project = Sdk.Fixtures.Exhaustive let config = @@ -27,8 +25,6 @@ let config = { packageName = None Text , emitSync = Some False , onUnsupported = Some OnUnsupported.Mode.Skip - , queryNameMappings = None (List PythonNameMapping.Query) - , customTypeNameMappings = None (List PythonNameMapping.CustomType) } in Sdk.Output.toFileMap (Gen.compile config project) diff --git a/src/Interpreters/CustomType.dhall b/src/Interpreters/CustomType.dhall index bf0241b..cdd259b 100644 --- a/src/Interpreters/CustomType.dhall +++ b/src/Interpreters/CustomType.dhall @@ -8,10 +8,6 @@ let ImportSet = ../Structures/ImportSet.dhall let CustomKind = ../Structures/CustomKind.dhall -let PythonNameMapping = ../Structures/PythonNameMapping.dhall - -let PythonNamespace = ../Structures/PythonNamespace.dhall - let PyIdent = ../Structures/PyIdent.dhall let MemberGen = ./Member.dhall @@ -20,9 +16,7 @@ let EnumModule = ../Templates/EnumModule.dhall let CompositeModule = ../Templates/CompositeModule.dhall -let Config = - { customTypeNameMappings : List PythonNameMapping.CustomType - } +let Config = {} let Input = Model.CustomType @@ -38,7 +32,6 @@ let Output = , kind : TypeKind , order : Natural , dependencies : List Natural - , moduleBindings : List PythonNamespace.Binding } let renderExtraImports = @@ -77,100 +70,14 @@ let renderExtraImports = ) # customLines -let moduleNamespace = - \(input : Input) -> - "custom type module for schema ${Text/show input.pgSchema}, type ${Text/show input.pgName}" - -let moduleBinding = - \(namespace : Text) -> - \(owner : Text) -> - \(name : Text) -> - { namespace - , owner - , name - , remediation = "Choose a different custom type name mapping target" - } - -let enumNamespaceBindings = - \(input : Input) -> - \(typeName : Text) -> - let namespace = moduleNamespace input - - in [ moduleBinding - namespace - "custom type class for schema ${Text/show input.pgSchema}, type ${Text/show input.pgName}" - typeName - , moduleBinding namespace "enum base import" "StrEnum" - ] - -let compositeNamespaceBindings = - \(input : Input) -> - \(typeName : Text) -> - \(imports : ImportSet.Type) -> - let namespace = moduleNamespace input - - let imported = - ( if imports.uuid - then [ moduleBinding namespace "UUID primitive import" "UUID" ] - else [] : List PythonNamespace.Binding - ) - # ( if imports.datetime - then [ moduleBinding namespace "datetime primitive import" "datetime" ] - else [] : List PythonNamespace.Binding - ) - # ( if imports.date - then [ moduleBinding namespace "date primitive import" "date" ] - else [] : List PythonNamespace.Binding - ) - # ( if imports.time - then [ moduleBinding namespace "time primitive import" "time" ] - else [] : List PythonNamespace.Binding - ) - # ( if imports.timedelta - then [ moduleBinding namespace "timedelta primitive import" "timedelta" ] - else [] : List PythonNamespace.Binding - ) - # ( if imports.decimal - then [ moduleBinding namespace "Decimal primitive import" "Decimal" ] - else [] : List PythonNamespace.Binding - ) - # ( if imports.jsonValue - then [ moduleBinding namespace "generated core import" "JsonValue" ] - else [] : List PythonNamespace.Binding - ) - - let customImports = - Prelude.List.map - ImportSet.CustomImport - PythonNamespace.Binding - ( \(custom : ImportSet.CustomImport) -> - moduleBinding - namespace - "custom dependency import \".${custom.moduleName}.${custom.className}\"" - custom.className - ) - imports.customTypes - - in [ moduleBinding - namespace - "custom type class for schema ${Text/show input.pgSchema}, type ${Text/show input.pgName}" - typeName - , moduleBinding namespace "dataclass decorator import" "dataclass" - ] - # imported - # customImports - let run = \(config : Config) -> \(lookup : CustomKind.Lookup) -> \(input : Input) -> let pythonName = - PythonNameMapping.resolveCustomType - config.customTypeNameMappings - { schema = input.pgSchema, name = input.pgName } - { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase - , pascalCase = PyIdent.pySafeName input.name.inPascalCase - } + { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase + , pascalCase = PyIdent.pySafeName input.name.inPascalCase + } let typeName = pythonName.pascalCase @@ -210,8 +117,6 @@ let run = , kind = TypeKind.Enum , order = identity.order , dependencies = [] : List Natural - , moduleBindings = - enumNamespaceBindings input typeName } , Composite = \(_ : CustomKind.Identity) -> @@ -306,11 +211,6 @@ let run = , kind = TypeKind.Composite , order = identity.order , dependencies - , moduleBindings = - compositeNamespaceBindings - input - typeName - combinedImports } , Enum = \(_ : CustomKind.Identity) -> diff --git a/src/Interpreters/Project.dhall b/src/Interpreters/Project.dhall index b4bf60d..e7d06fb 100644 --- a/src/Interpreters/Project.dhall +++ b/src/Interpreters/Project.dhall @@ -28,10 +28,6 @@ let FacadeModule = ../Templates/FacadeModule.dhall let OnUnsupported = ../Structures/OnUnsupported.dhall -let PythonNameMapping = ../Structures/PythonNameMapping.dhall - -let PythonNamespace = ../Structures/PythonNamespace.dhall - let Report = { path : List Text, message : Text } -- The generator's public Config: every field is independently Optional, so a @@ -43,471 +39,21 @@ let Config = { packageName : Optional Text , emitSync : Optional Bool , onUnsupported : Optional OnUnsupported.Mode - , queryNameMappings : Optional (List PythonNameMapping.Query) - , customTypeNameMappings : Optional (List PythonNameMapping.CustomType) } --- The root resolves every public option once. Query receives only emitSync and --- its mappings, while CustomType receives only its mappings. Lower interpreters --- use empty configs except for Result's caller-supplied rowClassName. +-- The root resolves every public option once. Lower interpreters use empty +-- configs except for Query's emitSync and Result's caller-supplied rowClassName. let ResolvedConfig = { packageName : Text , importName : Text , emitSync : Bool , onUnsupported : OnUnsupported.Mode - , queryNameMappings : List PythonNameMapping.Query - , customTypeNameMappings : List PythonNameMapping.CustomType } let Input = Model.Project let Output = Lude.Files.Type -let QueryMappingState = - { seen : List Text, error : Optional Report } - -let CustomTypeMappingState = - { seen : List PythonNameMapping.CustomTypeSource - , error : Optional Report - } - -let CustomTypeIdentity = - { contractName : Text, source : PythonNameMapping.CustomTypeSource } - -let CustomTypeIdentityState = - { seen : List CustomTypeIdentity, error : Optional Report } - -let finishMappingValidation = - \(stateError : Optional Report) -> - Prelude.Optional.fold - Report - stateError - (Lude.Compiled.Type {}) - (\(report : Report) -> (Lude.Compiled.Type {}).Err report) - (Lude.Compiled.ok {} {=}) - -let validateCustomTypeIdentities = - \(customTypes : List Model.CustomType) -> - let step = - \(customType : Model.CustomType) -> - \(state : CustomTypeIdentityState) -> - Prelude.Optional.fold - Report - state.error - CustomTypeIdentityState - (\(_ : Report) -> state) - ( let contractName = customType.name.inSnakeCase - - let previous = - List/fold - CustomTypeIdentity - state.seen - (Optional PythonNameMapping.CustomTypeSource) - ( \(identity : CustomTypeIdentity) -> - \(found : Optional PythonNameMapping.CustomTypeSource) -> - if Text/equal - identity.contractName - contractName - then Some identity.source - else found - ) - (None PythonNameMapping.CustomTypeSource) - - let error = - Prelude.Optional.fold - PythonNameMapping.CustomTypeSource - previous - (Optional Report) - ( \(source : PythonNameMapping.CustomTypeSource) -> - Some - { path = [ "customTypes", contractName ] - , message = - "Ambiguous unqualified custom type identity " - ++ Text/show contractName - ++ ": schema " - ++ Text/show source.schema - ++ ", type " - ++ Text/show source.name - ++ " and schema " - ++ Text/show customType.pgSchema - ++ ", type " - ++ Text/show customType.pgName - ++ " share one contract Name. The upstream contract must preserve a distinct schema-qualified custom identifier; Python mappings cannot recover it" - } - ) - (None Report) - - let identity = - { contractName - , source = - { schema = customType.pgSchema - , name = customType.pgName - } - } - - in { seen = state.seen # [ identity ], error } - ) - - let state = - List/fold - Model.CustomType - customTypes - CustomTypeIdentityState - step - { seen = [] : List CustomTypeIdentity - , error = None Report - } - - in finishMappingValidation state.error - -let validateQueryMappings = - \(mappings : List PythonNameMapping.Query) -> - \(queries : List Model.Query) -> - let step = - \(mapping : PythonNameMapping.Query) -> - \(state : QueryMappingState) -> - Prelude.Optional.fold - Report - state.error - QueryMappingState - (\(_ : Report) -> state) - ( let duplicate = - Prelude.List.any - Text - (\(source : Text) -> Text/equal source mapping.source) - state.seen - - let known = - Prelude.List.any - Model.Query - ( \(query : Model.Query) -> - Text/equal query.name.inSnakeCase mapping.source - ) - queries - - let snakeIsExact = - PyIdent.isSnakeIdentifier mapping.target.snakeCase - && Text/equal - (PyIdent.querySafeName mapping.target.snakeCase) - mapping.target.snakeCase - - let pascalIsExact = - PyIdent.isPascalIdentifier mapping.target.pascalCase - && Text/equal - (PyIdent.pySafeName mapping.target.pascalCase) - mapping.target.pascalCase - - let error = - if duplicate - then Some - { path = - [ "config" - , "queryNameMappings" - , mapping.source - ] - , message = - "Duplicate query name mapping source \"${mapping.source}\"" - } - else if Prelude.Bool.not known - then Some - { path = - [ "config" - , "queryNameMappings" - , mapping.source - ] - , message = - "Unknown query name mapping source \"${mapping.source}\"" - } - else if Prelude.Bool.not snakeIsExact - then Some - { path = - [ "config" - , "queryNameMappings" - , mapping.source - , "target" - , "snakeCase" - ] - , message = - "Mapped query snakeCase must start with a lowercase ASCII letter, then contain only lowercase ASCII letters, digits, or underscores, and must not be reserved" - } - else if Prelude.Bool.not pascalIsExact - then Some - { path = - [ "config" - , "queryNameMappings" - , mapping.source - , "target" - , "pascalCase" - ] - , message = - "Mapped query pascalCase must start with an uppercase ASCII letter, then contain only ASCII letters or digits, and must not be reserved" - } - else None Report - - in { seen = state.seen # [ mapping.source ], error } - ) - - let state = - List/fold - PythonNameMapping.Query - mappings - QueryMappingState - step - { seen = [] : List Text, error = None Report } - - in finishMappingValidation state.error - -let validateCustomTypeMappings = - \(mappings : List PythonNameMapping.CustomType) -> - \(customTypes : List Model.CustomType) -> - let sameSource = - \(left : PythonNameMapping.CustomTypeSource) -> - \(right : PythonNameMapping.CustomTypeSource) -> - Text/equal left.schema right.schema - && Text/equal left.name right.name - - let step = - \(mapping : PythonNameMapping.CustomType) -> - \(state : CustomTypeMappingState) -> - Prelude.Optional.fold - Report - state.error - CustomTypeMappingState - (\(_ : Report) -> state) - ( let duplicate = - Prelude.List.any - PythonNameMapping.CustomTypeSource - (sameSource mapping.source) - state.seen - - let known = - Prelude.List.any - Model.CustomType - ( \(customType : Model.CustomType) -> - Text/equal - customType.pgSchema - mapping.source.schema - && Text/equal - customType.pgName - mapping.source.name - ) - customTypes - - let snakeIsExact = - PyIdent.isSnakeIdentifier mapping.target.snakeCase - && Text/equal - ( PyIdent.typeModuleSafeName - mapping.target.snakeCase - ) - mapping.target.snakeCase - - let pascalIsExact = - PyIdent.isPascalIdentifier mapping.target.pascalCase - && Text/equal - (PyIdent.pySafeName mapping.target.pascalCase) - mapping.target.pascalCase - - let source = - "${mapping.source.schema}.${mapping.source.name}" - - let error = - if duplicate - then Some - { path = - [ "config" - , "customTypeNameMappings" - , source - ] - , message = - "Duplicate custom type name mapping source \"${source}\"" - } - else if Prelude.Bool.not known - then Some - { path = - [ "config" - , "customTypeNameMappings" - , source - ] - , message = - "Unknown custom type name mapping source \"${source}\"" - } - else if Prelude.Bool.not snakeIsExact - then Some - { path = - [ "config" - , "customTypeNameMappings" - , source - , "target" - , "snakeCase" - ] - , message = - "Mapped custom type snakeCase must start with a lowercase ASCII letter, then contain only lowercase ASCII letters, digits, or underscores, and must not be reserved" - } - else if Prelude.Bool.not pascalIsExact - then Some - { path = - [ "config" - , "customTypeNameMappings" - , source - , "target" - , "pascalCase" - ] - , message = - "Mapped custom type pascalCase must start with an uppercase ASCII letter, then contain only ASCII letters or digits, and must not be reserved" - } - else None Report - - in { seen = state.seen # [ mapping.source ], error } - ) - - let state = - List/fold - PythonNameMapping.CustomType - mappings - CustomTypeMappingState - step - { seen = [] : List PythonNameMapping.CustomTypeSource - , error = None Report - } - - in finishMappingValidation state.error - -let validateLocalNamespaces = - \(queries : List Model.Query) -> - \(customTypes : List Model.CustomType) -> - let validateQuery = - \(query : Model.Query) -> - let querySource = Text/show query.name.inSnakeCase - - let parameterBindings = - Prelude.List.map - Model.Member - PythonNamespace.Binding - ( \(parameter : Model.Member) -> - { namespace = - "parameters for query ${querySource}" - , owner = - "query parameter ${Text/show parameter.pgName}" - , name = - PyIdent.parameterSafeName - parameter.name.inSnakeCase - , remediation = "Rename one SQL placeholder" - } - ) - query.params - - let resultColumns = - merge - { Void = [] : List Model.Member - , RowsAffected = [] : List Model.Member - , Rows = - \(rows : Model.ResultRows) -> - Prelude.NonEmpty.toList - Model.Member - rows.columns - } - query.result - - let resultBindings = - Prelude.List.map - Model.Member - PythonNamespace.Binding - ( \(column : Model.Member) -> - { namespace = - "result fields for query ${querySource}" - , owner = - "result column ${Text/show column.pgName}" - , name = - PyIdent.pySafeName column.name.inSnakeCase - , remediation = "Rename one SQL result alias" - } - ) - resultColumns - - in PythonNamespace.validate - (parameterBindings # resultBindings) - - let validateCustomType = - \(customType : Model.CustomType) -> - let customSource = - "schema ${Text/show customType.pgSchema}, type ${Text/show customType.pgName}" - - let bindings = - merge - { Composite = - \(members : List Model.Member) -> - Prelude.List.map - Model.Member - PythonNamespace.Binding - ( \(member : Model.Member) -> - { namespace = - "fields for custom type ${customSource}" - , owner = - "composite field ${Text/show member.pgName}" - , name = - PyIdent.pySafeName - member.name.inSnakeCase - , remediation = - "Rename one PostgreSQL composite field" - } - ) - members - , Enum = - \(variants : List Model.EnumVariant) -> - Prelude.List.map - Model.EnumVariant - PythonNamespace.Binding - ( \(variant : Model.EnumVariant) -> - { namespace = - "enum members for custom type ${customSource}" - , owner = - "enum label ${Text/show variant.pgName}" - , name = - PyIdent.pySafeName - variant.name.inScreamingSnakeCase - , remediation = - "Rename one PostgreSQL enum label" - } - ) - variants - , Domain = - \(_ : Model.Value) -> - [] : List PythonNamespace.Binding - } - customType.definition - - in PythonNamespace.validate bindings - - let queryValidation = - Lude.Compiled.map - (List {}) - {} - (\(_ : List {}) -> {=}) - ( Lude.Compiled.traverseList - Model.Query - {} - validateQuery - queries - ) - - let customTypeValidation = - Lude.Compiled.map - (List {}) - {} - (\(_ : List {}) -> {=}) - ( Lude.Compiled.traverseList - Model.CustomType - {} - validateCustomType - customTypes - ) - - in Lude.Compiled.flatMap - {} - {} - (\(_ : {}) -> customTypeValidation) - queryValidation - -- Header of every emitted .py file. The marker truthfully warns that another -- generation replaces manual changes. The SPDX pair (REUSE convention) -- licenses the emitted file itself as MIT-0. @@ -537,7 +83,6 @@ let ResolvedCustomType = { value : Model.CustomType, lookupEntry : LookupEntry } let resolveCustomTypes = - \(nameMappings : List PythonNameMapping.CustomType) -> \(customTypes : List Model.CustomType) -> Prelude.List.map IndexedCustomType @@ -546,14 +91,10 @@ let resolveCustomTypes = let customType = entry.value let pythonName = - PythonNameMapping.resolveCustomType - nameMappings - { schema = customType.pgSchema, name = customType.pgName } - { snakeCase = - PyIdent.typeModuleSafeName - customType.name.inSnakeCase - , pascalCase = PyIdent.pySafeName customType.name.inPascalCase - } + { snakeCase = + PyIdent.typeModuleSafeName customType.name.inSnakeCase + , pascalCase = PyIdent.pySafeName customType.name.inPascalCase + } let identity = { className = pythonName.pascalCase @@ -699,161 +240,6 @@ let registrationOrder = [ "register_types" ] "Unresolved or cyclic custom type dependencies: ${unresolved}" -let validateProjectNamespaces = - \(config : ResolvedConfig) -> - \(queries : List QueryGen.Output) -> - \(customTypes : List CustomTypeGen.Output) -> - let mappingRemediation = - "Rename the source or configure a typed name mapping" - - let queryOwner = - \(query : QueryGen.Output) -> - "query \"${query.sourceName}\" (${query.sourcePath})" - - let queryModuleBindings = - Prelude.List.map - QueryGen.Output - PythonNamespace.Binding - ( \(query : QueryGen.Output) -> - { namespace = "generated statement modules" - , owner = queryOwner query - , name = query.functionName - , remediation = mappingRemediation - } - ) - queries - - let typeModuleBindings = - Prelude.List.map - CustomTypeGen.Output - PythonNamespace.Binding - ( \(customType : CustomTypeGen.Output) -> - { namespace = "generated custom type modules" - , owner = - "custom type \"${customType.pgSchema}.${customType.pgName}\"" - , name = customType.moduleName - , remediation = mappingRemediation - } - ) - customTypes - - let coreFacadeBindings = - [ { namespace = "package facade" - , owner = "generated core symbol JsonValue" - , name = "JsonValue" - , remediation = mappingRemediation - } - , { namespace = "package facade" - , owner = "generated core symbol NoRowError" - , name = "NoRowError" - , remediation = mappingRemediation - } - ] : List PythonNamespace.Binding - - let syncFacadeBindings = - if config.emitSync - then [ { namespace = "package facade" - , owner = "generated sync facade" - , name = "sync" - , remediation = mappingRemediation - } - ] - else [] : List PythonNamespace.Binding - - let registrationFacadeBindings = - if Prelude.List.null CustomTypeGen.Output customTypes - then [] : List PythonNamespace.Binding - else [ { namespace = "package facade" - , owner = "generated type registration function" - , name = "register_types" - , remediation = mappingRemediation - } - ] - - let queryFacadeBindings = - Prelude.List.concatMap - QueryGen.Output - PythonNamespace.Binding - ( \(query : QueryGen.Output) -> - let functionBinding = - { namespace = "package facade" - , owner = queryOwner query - , name = query.functionName - , remediation = mappingRemediation - } - - let rowBindings = - Prelude.Optional.fold - Text - query.rowClassName - (List PythonNamespace.Binding) - ( \(rowClassName : Text) -> - [ { namespace = "package facade" - , owner = - "Row class from ${queryOwner query}" - , name = rowClassName - , remediation = mappingRemediation - } - ] - ) - ([] : List PythonNamespace.Binding) - - in [ functionBinding ] # rowBindings - ) - queries - - let typeFacadeBindings = - Prelude.List.map - CustomTypeGen.Output - PythonNamespace.Binding - ( \(customType : CustomTypeGen.Output) -> - { namespace = "package facade" - , owner = - "custom type \"${customType.pgSchema}.${customType.pgName}\"" - , name = customType.typeName - , remediation = mappingRemediation - } - ) - customTypes - - -- validate freezes a right-folded state, so groups are supplied in - -- reverse diagnostic priority. Module/file collisions remain the - -- primary cause when the same pair would also collide in a facade. - let projectValidation = - PythonNamespace.validate - ( typeFacadeBindings - # queryFacadeBindings - # registrationFacadeBindings - # syncFacadeBindings - # coreFacadeBindings - # typeModuleBindings - # queryModuleBindings - ) - - -- A module-name collision is fatal in Fail and Skip alike. These - -- bindings travel through CustomType.Output so Skip's support probe - -- cannot mistake a naming error for an unsupported PostgreSQL shape. - let moduleValidation = - Lude.Compiled.map - (List {}) - {} - (\(_ : List {}) -> {=}) - ( Lude.Compiled.traverseList - CustomTypeGen.Output - {} - ( \(customType : CustomTypeGen.Output) -> - PythonNamespace.validate - customType.moduleBindings - ) - customTypes - ) - - in Lude.Compiled.flatMap - {} - {} - (\(_ : {}) -> projectValidation) - moduleValidation - let combineOutputs = \(config : ResolvedConfig) -> \(input : Input) -> @@ -1101,39 +487,15 @@ let run = (\(m : OnUnsupported.Mode) -> m) OnUnsupported.Mode.Fail - let queryNameMappings = - Prelude.Optional.fold - (List PythonNameMapping.Query) - config.queryNameMappings - (List PythonNameMapping.Query) - (\(mappings : List PythonNameMapping.Query) -> mappings) - ([] : List PythonNameMapping.Query) - - let customTypeNameMappings = - Prelude.Optional.fold - (List PythonNameMapping.CustomType) - config.customTypeNameMappings - (List PythonNameMapping.CustomType) - (\(mappings : List PythonNameMapping.CustomType) -> mappings) - ([] : List PythonNameMapping.CustomType) - let importName = Prelude.Text.replace "-" "_" packageName let resolvedConfig : ResolvedConfig - = { packageName - , importName - , emitSync - , onUnsupported - , queryNameMappings - , customTypeNameMappings - } + = { packageName, importName, emitSync, onUnsupported } - let queryConfig = - resolvedConfig.{ emitSync, queryNameMappings } + let queryConfig = resolvedConfig.{ emitSync } - let customTypeConfig = - resolvedConfig.{ customTypeNameMappings } + let customTypeConfig = {=} let skip = merge { Fail = False, Skip = True } resolvedConfig.onUnsupported @@ -1163,10 +525,7 @@ let run = } (CustomTypeGen.run customTypeConfig candidateLookup ct) - let resolvedCustomTypes = - resolveCustomTypes - resolvedConfig.customTypeNameMappings - input.customTypes + let resolvedCustomTypes = resolveCustomTypes input.customTypes -- Nested custom support requires transitive closure: after one type is -- removed, composites depending on it must be reconsidered against the @@ -1295,66 +654,19 @@ let run = let combined : Lude.Compiled.Type Output - = Lude.Compiled.flatMap + = Lude.Compiled.map CombinedInputs Output ( \(compiled : CombinedInputs) -> - Lude.Compiled.map - {} - Output - ( \(_ : {}) -> - combineOutputs - resolvedConfig - input - compiled.queries - compiled.customTypes - compiled.registrationTypes - ) - ( validateProjectNamespaces - resolvedConfig - compiled.queries - compiled.customTypes - ) + combineOutputs + resolvedConfig + input + compiled.queries + compiled.customTypes + compiled.registrationTypes ) compiledInputs - let mappingsValid = - Lude.Compiled.flatMap - {} - {} - ( \(_ : {}) -> - Lude.Compiled.flatMap - {} - {} - ( \(_ : {}) -> - validateCustomTypeMappings - resolvedConfig.customTypeNameMappings - input.customTypes - ) - ( validateQueryMappings - resolvedConfig.queryNameMappings - input.queries - ) - ) - (validateCustomTypeIdentities input.customTypes) - - let mappingsAndLocalsValid = - Lude.Compiled.flatMap - {} - {} - ( \(_ : {}) -> - validateLocalNamespaces - effectiveQueries - effectiveCustomTypes - ) - mappingsValid - - in Lude.Compiled.flatMap - {} - Output - ( \(_ : {}) -> - Lude.Compiled.appendWarnings Output skipWarnings combined - ) - mappingsAndLocalsValid + in Lude.Compiled.appendWarnings Output skipWarnings combined in Sdk.Sigs.interpreter Config Input Output run diff --git a/src/Interpreters/Query.dhall b/src/Interpreters/Query.dhall index b4eb4a5..30d80b6 100644 --- a/src/Interpreters/Query.dhall +++ b/src/Interpreters/Query.dhall @@ -12,8 +12,6 @@ let PyIdent = ../Structures/PyIdent.dhall let Surface = ../Structures/Surface.dhall -let PythonNameMapping = ../Structures/PythonNameMapping.dhall - let ResultModule = ./Result.dhall let QueryFragmentsModule = ./QueryFragments.dhall @@ -22,10 +20,7 @@ let ParamsMember = ./ParamsMember.dhall let StatementModule = ../Templates/StatementModule.dhall -let Config = - { emitSync : Bool - , queryNameMappings : List PythonNameMapping.Query - } +let Config = { emitSync : Bool } let Compiled = Lude.Compiled @@ -129,12 +124,9 @@ let run = \(lookup : CustomKind.Lookup) -> \(input : Input) -> let pythonName = - PythonNameMapping.resolveQuery - config.queryNameMappings - input.name.inSnakeCase - { snakeCase = PyIdent.querySafeName input.name.inSnakeCase - , pascalCase = input.name.inPascalCase - } + { snakeCase = PyIdent.querySafeName input.name.inSnakeCase + , pascalCase = input.name.inPascalCase + } let rowClassName = pythonName.pascalCase ++ "Row" diff --git a/src/Structures/PyIdent.dhall b/src/Structures/PyIdent.dhall index e769a35..822f44d 100644 --- a/src/Structures/PyIdent.dhall +++ b/src/Structures/PyIdent.dhall @@ -1,5 +1,3 @@ -let Prelude = ../Deps/Prelude.dhall - -- pgn passes identifier names through verbatim (the keyword-param fixture proves a -- column or placeholder may be spelled as a Python reserved word), so any name that -- becomes a Python identifier in the emitted code must be sanitized. Shared so the @@ -129,64 +127,6 @@ let querySafeName = let typeModuleSafeName = \(name : Text) -> sanitizeAgainst (pythonKeywords # moduleMetadataNames) name -let asciiLetters = - [ "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m" - , "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z" - , "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M" - , "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z" - ] - -let digits = [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" ] - -let containsOnly = - \(allowed : List Text) -> - \(name : Text) -> - let residue = - List/fold - Text - allowed - Text - (\(character : Text) -> \(rest : Text) -> Text/replace character "" rest) - name - - in Text/equal residue "" - --- The string is already limited to identifier characters before this check, so --- its Text/show value contains a quote only at each end. Replacing `"` can --- therefore match only the first character and gives us a safe head predicate --- without a non-standard regex builtin. -let startsWithOneOf = - \(allowed : List Text) -> - \(name : Text) -> - let shown = Text/show name - - in Prelude.List.any - Text - ( \(character : Text) -> - Prelude.Bool.not - ( Text/equal - (Text/replace ("\"" ++ character) "" shown) - shown - ) - ) - allowed - -let isSnakeIdentifier = - \(name : Text) -> - let letters = Prelude.List.take 26 Text asciiLetters - - in Prelude.Bool.not (Text/equal name "") - && containsOnly (letters # digits # [ "_" ]) name - && startsWithOneOf letters name - -let isPascalIdentifier = - \(name : Text) -> - let uppercaseLetters = Prelude.List.drop 26 Text asciiLetters - - in Prelude.Bool.not (Text/equal name "") - && containsOnly (asciiLetters # digits) name - && startsWithOneOf uppercaseLetters name - in { pythonKeywords , moduleMetadataNames , sanitizeAgainst @@ -196,6 +136,4 @@ in { pythonKeywords , parameterSafeName , querySafeName , typeModuleSafeName - , isSnakeIdentifier - , isPascalIdentifier } diff --git a/src/Structures/PythonNameMapping.dhall b/src/Structures/PythonNameMapping.dhall deleted file mode 100644 index a955517..0000000 --- a/src/Structures/PythonNameMapping.dhall +++ /dev/null @@ -1,48 +0,0 @@ -let Prelude = ../Deps/Prelude.dhall - -let PythonName = { snakeCase : Text, pascalCase : Text } - -let Query = { source : Text, target : PythonName } - -let CustomTypeSource = { schema : Text, name : Text } - -let CustomType = { source : CustomTypeSource, target : PythonName } - -let resolveQuery = - \(mappings : List Query) -> - \(source : Text) -> - \(defaultName : PythonName) -> - List/fold - Query - mappings - PythonName - ( \(mapping : Query) -> - \(resolved : PythonName) -> - if Text/equal mapping.source source then mapping.target else resolved - ) - defaultName - -let resolveCustomType = - \(mappings : List CustomType) -> - \(source : CustomTypeSource) -> - \(defaultName : PythonName) -> - List/fold - CustomType - mappings - PythonName - ( \(mapping : CustomType) -> - \(resolved : PythonName) -> - if Text/equal mapping.source.schema source.schema - && Text/equal mapping.source.name source.name - then mapping.target - else resolved - ) - defaultName - -in { PythonName - , Query - , CustomTypeSource - , CustomType - , resolveQuery - , resolveCustomType - } diff --git a/src/Structures/PythonNamespace.dhall b/src/Structures/PythonNamespace.dhall deleted file mode 100644 index 28a9fcc..0000000 --- a/src/Structures/PythonNamespace.dhall +++ /dev/null @@ -1,97 +0,0 @@ -let Lude = ../Deps/Lude.dhall - -let Prelude = ../Deps/Prelude.dhall - -let Binding = - { namespace : Text - , owner : Text - , name : Text - , remediation : Text - } - -let Collision = - { namespace : Text - , firstOwner : Text - , secondOwner : Text - , name : Text - , remediation : Text - } - -let State = { seen : List Binding, collision : Optional Collision } - -let validate = - \(bindings : List Binding) -> - let step = - \(binding : Binding) -> - \(state : State) -> - Prelude.Optional.fold - Collision - state.collision - State - (\(_ : Collision) -> state) - ( let previousOwner = - List/fold - Binding - state.seen - (Optional Text) - ( \(previous : Binding) -> - \(found : Optional Text) -> - if Text/equal - previous.namespace - binding.namespace - && Text/equal previous.name binding.name - then Some previous.owner - else found - ) - (None Text) - - let collision = - Prelude.Optional.fold - Text - previousOwner - (Optional Collision) - ( \(owner : Text) -> - Some - { namespace = binding.namespace - , firstOwner = owner - , secondOwner = binding.owner - , name = binding.name - , remediation = binding.remediation - } - ) - (None Collision) - - in { seen = state.seen # [ binding ], collision } - ) - - let result = - List/fold - Binding - bindings - State - step - { seen = [] : List Binding, collision = None Collision } - - in Prelude.Optional.fold - Collision - result.collision - (Lude.Compiled.Type {}) - ( \(collision : Collision) -> - Lude.Compiled.report - {} - [ "pythonNames", collision.namespace, collision.name ] - ( "Generated Python name collision in " - ++ collision.namespace - ++ ": " - ++ collision.firstOwner - ++ " and " - ++ collision.secondOwner - ++ " both resolve to \"" - ++ collision.name - ++ "\". " - ++ collision.remediation - ) - ) - (Lude.Compiled.ok {} {=}) - -in { Binding, validate } diff --git a/src/package.dhall b/src/package.dhall index a820150..67ff3c1 100644 --- a/src/package.dhall +++ b/src/package.dhall @@ -2,8 +2,6 @@ let Sdk = ./Deps/Sdk.dhall let OnUnsupported = ./Structures/OnUnsupported.dhall -let PythonNameMapping = ./Structures/PythonNameMapping.dhall - let ProjectInterpreter = ./Interpreters/Project.dhall -- User-facing config for this generator. Async output is always emitted; @@ -13,14 +11,11 @@ let ProjectInterpreter = ./Interpreters/Project.dhall -- query or custom type hits a PG shape the generator cannot render; see -- Structures/OnUnsupported.dhall. All fields are Optional so a project may -- omit the whole config block or any subset of its keys; --- Interpreters/Project.dhall's `run` supplies the defaults. Typed name mappings --- provide explicit whole-entity renames after collision checks. +-- Interpreters/Project.dhall's `run` supplies the defaults. let Config = { packageName : Optional Text , emitSync : Optional Bool , onUnsupported : Optional OnUnsupported.Mode - , queryNameMappings : Optional (List PythonNameMapping.Query) - , customTypeNameMappings : Optional (List PythonNameMapping.CustomType) } : Type let Config/default @@ -28,8 +23,6 @@ let Config/default = { packageName = None Text , emitSync = None Bool , onUnsupported = None OnUnsupported.Mode - , queryNameMappings = None (List PythonNameMapping.Query) - , customTypeNameMappings = None (List PythonNameMapping.CustomType) } in Sdk.Sigs.generator Config Config/default ProjectInterpreter.run diff --git a/tests/test_python_name_collisions.py b/tests/test_python_name_collisions.py deleted file mode 100644 index 8503264..0000000 --- a/tests/test_python_name_collisions.py +++ /dev/null @@ -1,692 +0,0 @@ -"""Real-pgn coverage for generated Python namespace collisions and mappings.""" - -from __future__ import annotations - -import ast -import importlib -import shutil -import subprocess -import sys -from collections.abc import Iterable, Mapping, Sequence -from pathlib import Path - -import pytest - -from tests._harness import SRC_DIR, run_pgn - -QueryMapping = tuple[str, str, str] -CustomTypeMapping = tuple[str, str, str, str] - - -def _project_yaml( - *, - query_mappings: Sequence[QueryMapping] = (), - custom_type_mappings: Sequence[CustomTypeMapping] = (), - on_unsupported: str | None = None, -) -> str: - lines = [ - "space: python-gen", - "name: python-name-collisions", - "version: 0.0.0", - "postgres: 18", - "artifacts:", - " python:", - " gen: ../src/package.dhall", - " config:", - " packageName: mapping-client", - " emitSync: true", - ] - - if on_unsupported is not None: - lines.append(f" onUnsupported: {on_unsupported}") - - if query_mappings: - lines.append(" queryNameMappings:") - for source, snake_case, pascal_case in query_mappings: - lines.extend( - [ - f" - source: {source}", - " target:", - f" snakeCase: {snake_case}", - f" pascalCase: {pascal_case}", - ] - ) - - if custom_type_mappings: - lines.append(" customTypeNameMappings:") - for schema, name, snake_case, pascal_case in custom_type_mappings: - lines.extend( - [ - " - source:", - f" schema: {schema}", - f" name: {name}", - " target:", - f" snakeCase: {snake_case}", - f" pascalCase: {pascal_case}", - ] - ) - - return "\n".join(lines) + "\n" - - -def _fresh_project( - tmp_path: Path, - *, - queries: Mapping[str, str], - migration: str = "SELECT 1;\n", - query_mappings: Sequence[QueryMapping] = (), - custom_type_mappings: Sequence[CustomTypeMapping] = (), - on_unsupported: str | None = None, -) -> Path: - root = tmp_path / "pygen" - _ = shutil.copytree(SRC_DIR, root / "src") - project = root / "project" - migrations = project / "migrations" - query_dir = project / "queries" - migrations.mkdir(parents=True) - query_dir.mkdir() - - _ = (migrations / "1.sql").write_text(migration) - for name, sql in queries.items(): - _ = (query_dir / name).write_text(sql) - _ = (project / "project1.pgn.yaml").write_text( - _project_yaml( - query_mappings=query_mappings, - custom_type_mappings=custom_type_mappings, - on_unsupported=on_unsupported, - ) - ) - return project - - -def _write_structural_scope_probe(project: Path, *, kind: str, duplicate_identity: bool) -> None: - root = project.parent - right_name = "leftName" if duplicate_identity else "rightName" - definitions = { - "composite": ( - "< Enum : List Model.EnumVariant | Composite : List Model.Member | Domain : Model.Value >" - ".Composite [ valueMember ]" - ), - "enum": ( - "< Enum : List Model.EnumVariant | Composite : List Model.Member | Domain : Model.Value >" - ".Enum [ readyVariant ]" - ), - } - definition = definitions[kind] - wrapper = f""" -let Model = ./Deps/Contract.dhall - -let Sdk = ./Deps/Sdk.dhall - -let OnUnsupported = ./Structures/OnUnsupported.dhall - -let PythonNameMapping = ./Structures/PythonNameMapping.dhall - -let Target = ./Interpreters/Project.dhall - -let Config = - {{ packageName : Optional Text - , emitSync : Optional Bool - , onUnsupported : Optional OnUnsupported.Mode - }} - -let Config/default = - {{ packageName = None Text - , emitSync = None Bool - , onUnsupported = None OnUnsupported.Mode - }} - -let valueName - : Model.Name - = {{ inCamelCase = "value" - , inPascalCase = "Value" - , inKebabCase = "value" - , inTrainCase = "Value" - , inScreamingKebabCase = "VALUE" - , inSnakeCase = "value" - , inCamelSnakeCase = "Value" - , inScreamingSnakeCase = "VALUE" - }} - -let leftName - : Model.Name - = valueName // {{ inCamelCase = "leftSource", inPascalCase = "LeftSource", inSnakeCase = "left_source" }} - -let rightName - : Model.Name - = valueName // {{ inCamelCase = "rightSource", inPascalCase = "RightSource", inSnakeCase = "right_source" }} - -let valueMember - : Model.Member - = {{ isNullable = False - , name = valueName - , pgName = "value" - , value = - {{ arraySettings = None Model.ArraySettings - , scalar = Model.Scalar.Primitive Model.Primitive.Int8 - }} - }} - -let readyVariant - : Model.EnumVariant - = {{ name = valueName // {{ inScreamingSnakeCase = "READY" }} - , pgName = "ready" - }} - -let leftType - : Model.CustomType - = {{ definition = {definition} - , name = leftName - , pgName = "c" - , pgSchema = "a.b" - }} - -let rightType - : Model.CustomType - = {{ definition = {definition} - , name = {right_name} - , pgName = "b.c" - , pgSchema = "a" - }} - -let targetConfig = - {{ packageName = Some "mapping-client" - , emitSync = Some False - , onUnsupported = Some OnUnsupported.Mode.Fail - , queryNameMappings = None (List PythonNameMapping.Query) - , customTypeNameMappings = - Some - [ {{ source = {{ schema = "a.b", name = "c" }} - , target = {{ snakeCase = "schema_dot_c", pascalCase = "SchemaDotC" }} - }} - , {{ source = {{ schema = "a", name = "b.c" }} - , target = {{ snakeCase = "type_dot_c", pascalCase = "TypeDotC" }} - }} - ] - }} - -let run = - \\(_ : Config) -> - \\(input : Model.Project) -> - Target.run - targetConfig - ( input - // {{ customTypes = [ leftType, rightType ] - , queries = [] : List Model.Query - }} - ) - -in Sdk.Sigs.generator Config Config/default run -""" - _ = (root / "src" / "structural-scope-probe.dhall").write_text(wrapper) - project_file = project / "project1.pgn.yaml" - _ = project_file.write_text( - project_file.read_text().replace("../src/package.dhall", "../src/structural-scope-probe.dhall") - ) - - -def _analyse(pgn_bin: str, pgn_admin_url: str, project: Path) -> None: - result = run_pgn(pgn_bin, pgn_admin_url, project, "analyse") - assert result.returncode == 0, f"pgn analyse rejected the collision fixture:\n{result.stdout}\n{result.stderr}" - - -def _diagnostic(result: subprocess.CompletedProcess[str]) -> str: - return f"{result.stdout}\n{result.stderr}" - - -def _assert_no_python_files(project: Path) -> None: - artifact = project / "artifacts" / "python" - assert not artifact.exists() or not any(artifact.rglob("*.py")) - - -def _assert_generation_fails( - pgn_bin: str, - pgn_admin_url: str, - project: Path, - expected: Iterable[str], -) -> None: - result = run_pgn(pgn_bin, pgn_admin_url, project, "generate") - diagnostic = _diagnostic(result) - assert result.returncode != 0, f"pgn generate unexpectedly succeeded:\n{diagnostic}" - for fragment in expected: - assert fragment in diagnostic, f"missing {fragment!r} in pgn diagnostic:\n{diagnostic}" - _assert_no_python_files(project) - - -def _package_dir(project: Path) -> Path: - package = project / "artifacts" / "python" / "src" / "mapping_client" - assert package.is_dir() - return package - - -def _clear_package_modules() -> None: - for name in list(sys.modules): - if name == "mapping_client" or name.startswith("mapping_client."): - del sys.modules[name] - - -def test_reserved_query_collision_fails_before_emission( - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project( - tmp_path, - queries={ - "sync.sql": "SELECT 1::int8 AS value\n", - "sync_query.sql": "SELECT 2::int8 AS value\n", - }, - ) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails( - pgn_bin, - pgn_admin_url, - project, - [ - "Generated Python name collision in generated statement modules", - 'query "sync"', - 'query "sync_query"', - 'both resolve to "sync_query"', - ], - ) - - -def test_json_value_custom_type_collision_fails_before_emission( - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project( - tmp_path, - migration="CREATE TYPE \"json_value\" AS ENUM ('ready');\n", - queries={"read_json_value.sql": "SELECT 'ready'::public.\"json_value\" AS value\n"}, - ) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails( - pgn_bin, - pgn_admin_url, - project, - [ - "Generated Python name collision in package facade", - "generated core symbol JsonValue", - 'custom type "public.json_value"', - 'both resolve to "JsonValue"', - ], - ) - - -def test_query_row_and_custom_type_collision_fails_before_emission( - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project( - tmp_path, - migration="CREATE TYPE foo_row AS ENUM ('ready');\n", - queries={"foo.sql": "SELECT 'ready'::foo_row AS value\n"}, - ) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails( - pgn_bin, - pgn_admin_url, - project, - [ - "Generated Python name collision in package facade", - 'Row class from query "foo"', - 'custom type "public.foo_row"', - 'both resolve to "FooRow"', - ], - ) - - -@pytest.mark.parametrize( - ("native_type", "native_value", "mapped_snake", "mapped_pascal", "on_unsupported"), - [ - ( - "uuid", - "'00000000-0000-0000-0000-000000000001'::uuid", - "custom_uuid", - "UUID", - None, - ), - ("numeric", "1.25::numeric", "custom_decimal", "Decimal", None), - ( - "uuid", - "'00000000-0000-0000-0000-000000000001'::uuid", - "custom_uuid", - "UUID", - "Skip", - ), - ], - ids=["uuid-fail", "decimal-fail", "uuid-skip"], -) -def test_mapped_custom_dependency_cannot_shadow_composite_import( - native_type: str, - native_value: str, - mapped_snake: str, - mapped_pascal: str, - on_unsupported: str | None, - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project( - tmp_path, - migration=( - "CREATE TYPE import_dependency AS ENUM ('ready');\n" - f"CREATE TYPE import_holder AS (native_value {native_type}, dependency import_dependency);\n" - ), - queries={ - "read_import_holder.sql": ( - f"SELECT ROW({native_value}, 'ready'::import_dependency)::import_holder AS value\n" - ) - }, - custom_type_mappings=[("public", "import_dependency", mapped_snake, mapped_pascal)], - on_unsupported=on_unsupported, - ) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails( - pgn_bin, - pgn_admin_url, - project, - [ - 'Generated Python name collision in custom type module for schema "public", type "import_holder"', - f"{mapped_pascal} primitive import", - "custom dependency import", - f'both resolve to "{mapped_pascal}"', - ], - ) - - -@pytest.mark.parametrize("kind", ["composite", "enum"]) -def test_custom_local_namespaces_keep_schema_and_name_structured( - kind: str, - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project(tmp_path, queries={"seed.sql": "SELECT 1::int8 AS value\n"}) - _write_structural_scope_probe(project, kind=kind, duplicate_identity=False) - _analyse(pgn_bin, pgn_admin_url, project) - - result = run_pgn(pgn_bin, pgn_admin_url, project, "generate") - diagnostic = _diagnostic(result) - assert result.returncode == 0, f"structured local namespace probe failed:\n{diagnostic}" - - package = _package_dir(project) - left_source = (package / "_generated" / "types" / "schema_dot_c.py").read_text() - right_source = (package / "_generated" / "types" / "type_dot_c.py").read_text() - expected = ( - ("class SchemaDotC:", "class TypeDotC:", "value: int") - if kind == "composite" - else ("class SchemaDotC(StrEnum):", "class TypeDotC(StrEnum):", 'READY = "ready"') - ) - assert expected[0] in left_source - assert expected[1] in right_source - assert expected[2] in left_source and expected[2] in right_source - for source in package.rglob("*.py"): - _ = ast.parse(source.read_text(), filename=str(source)) - - -def test_duplicate_unqualified_custom_contract_identity_fails_loudly( - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project(tmp_path, queries={"seed.sql": "SELECT 1::int8 AS value\n"}) - _write_structural_scope_probe(project, kind="composite", duplicate_identity=True) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails( - pgn_bin, - pgn_admin_url, - project, - [ - 'Ambiguous unqualified custom type identity "left_source"', - 'schema "a.b", type "c"', - 'schema "a", type "b.c"', - "upstream contract must preserve a distinct schema-qualified custom identifier", - "Python mappings cannot recover it", - ], - ) - - -def test_typed_mappings_propagate_through_generated_package( - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project( - tmp_path, - migration=( - "CREATE TYPE \"json_value\" AS ENUM ('ready');\n" - "CREATE TYPE mapped_inner AS (value int8);\n" - "CREATE TYPE mapped_outer AS (child mapped_inner);\n" - ), - queries={ - "sync.sql": "SELECT 1::int8 AS value\n", - "sync_query.sql": "SELECT 2::int8 AS value\n", - "read_json_value.sql": "SELECT 'ready'::public.\"json_value\" AS value\n", - "read_mapped_outer.sql": ("SELECT ROW(ROW(7::int8)::mapped_inner)::mapped_outer AS value\n"), - }, - query_mappings=[("sync_query", "api_v2", "ApiV2")], - custom_type_mappings=[ - ("public", "json_value", "pg_json_value", "PgJsonValue"), - ("public", "mapped_inner", "mapped_inner_v2", "MappedInnerV2"), - ], - ) - _analyse(pgn_bin, pgn_admin_url, project) - result = run_pgn(pgn_bin, pgn_admin_url, project, "generate") - diagnostic = _diagnostic(result) - assert result.returncode == 0, f"mapped pgn generate failed:\n{diagnostic}" - - package = _package_dir(project) - statements = package / "_generated" / "statements" - assert {path.name for path in statements.glob("*.py")} == { - "__init__.py", - "api_v2.py", - "read_json_value.py", - "read_mapped_outer.py", - "sync_query.py", - } - api_source = (statements / "api_v2.py").read_text() - assert "class ApiV2Row:" in api_source - assert "async def api_v2(" in api_source - assert "def api_v2_sync(" in api_source - - type_source = (package / "_generated" / "types" / "pg_json_value.py").read_text() - assert "class PgJsonValue(StrEnum):" in type_source - inner_source = (package / "_generated" / "types" / "mapped_inner_v2.py").read_text() - assert "class MappedInnerV2:" in inner_source - outer_source = (package / "_generated" / "types" / "mapped_outer.py").read_text() - assert "from .mapped_inner_v2 import MappedInnerV2" in outer_source - assert "child: MappedInnerV2" in outer_source - json_statement = (statements / "read_json_value.py").read_text() - assert "value: _db_types.PgJsonValue" in json_statement - register_source = (package / "_generated" / "_register.py").read_text() - assert "_db_types.PgJsonValue" in register_source - assert "_db_types.MappedInnerV2" in register_source - assert register_source.index('"public.mapped_inner"') < register_source.index('"public.mapped_outer"') - assert "public.json_value" in register_source - - facade_source = (package / "__init__.py").read_text() - assert facade_source.index("statements.sync_query") < facade_source.index("statements.api_v2") - assert facade_source.index("types.pg_json_value") < facade_source.index("types.mapped_inner_v2") - assert facade_source.index('"PgJsonValue"') < facade_source.index('"MappedInnerV2"') - types_init_source = (package / "_generated" / "types" / "__init__.py").read_text() - assert types_init_source.index(".pg_json_value import") < types_init_source.index(".mapped_inner_v2 import") - - for source in package.rglob("*.py"): - _ = ast.parse(source.read_text(), filename=str(source)) - - ruff = subprocess.run( - [ - "ruff", - "check", - "--config", - str(SRC_DIR.parent / "pyproject.toml"), - str(package), - ], - capture_output=True, - text=True, - ) - assert ruff.returncode == 0, f"mapped generated package failed Ruff:\n{ruff.stdout}\n{ruff.stderr}" - - generated_src = package.parent - sys.path.insert(0, str(generated_src)) - _clear_package_modules() - try: - client = importlib.import_module("mapping_client") - sync_client = importlib.import_module("mapping_client.sync") - assert client.api_v2.__name__ == "api_v2" - assert sync_client.api_v2.__name__ == "api_v2_sync" - assert client.ApiV2Row is sync_client.ApiV2Row - assert client.PgJsonValue is sync_client.PgJsonValue - assert client.MappedInnerV2 is sync_client.MappedInnerV2 - assert client.MappedOuter is sync_client.MappedOuter - assert client.JsonValue is not client.PgJsonValue - finally: - _clear_package_modules() - sys.path.remove(str(generated_src)) - - -@pytest.mark.parametrize( - ("queries", "mappings", "expected"), - [ - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("alpha", "api_v2", "ApiV2"), ("alpha", "other_api", "OtherApi")], - ['Duplicate query name mapping source "alpha"'], - ), - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("missing", "api_v2", "ApiV2")], - ['Unknown query name mapping source "missing"'], - ), - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("alpha", "2api", "ApiV2")], - ["Mapped query snakeCase must start with a lowercase ASCII letter"], - ), - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("alpha", "api_v2", "2Api")], - ["Mapped query pascalCase must start with an uppercase ASCII letter"], - ), - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("alpha", "api_v2", "apiV2")], - ["Mapped query pascalCase must start with an uppercase ASCII letter"], - ), - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("alpha", "__init__", "Init")], - ["Mapped query snakeCase must start with a lowercase ASCII letter"], - ), - ( - {"alpha.sql": "SELECT 1::int8 AS value\n"}, - [("alpha", "__all__", "All")], - ["Mapped query snakeCase must start with a lowercase ASCII letter"], - ), - ( - { - "alpha.sql": "SELECT 1::int8 AS value\n", - "beta.sql": "SELECT 2::int8 AS value\n", - }, - [("beta", "alpha", "BetaResult")], - [ - "Generated Python name collision in generated statement modules", - 'query "alpha"', - 'query "beta"', - 'both resolve to "alpha"', - ], - ), - ], - ids=[ - "duplicate-source", - "unknown-source", - "leading-digit-snake", - "leading-digit-pascal", - "lowercase-pascal", - "init-module", - "all-facade", - "mapped-final-collision", - ], -) -def test_invalid_query_mappings_fail_before_emission( - queries: Mapping[str, str], - mappings: Sequence[QueryMapping], - expected: Sequence[str], - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project(tmp_path, queries=queries, query_mappings=mappings) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails(pgn_bin, pgn_admin_url, project, expected) - - -@pytest.mark.parametrize( - ("mappings", "expected"), - [ - ( - [ - ("public", "json_value", "pg_json_value", "PgJsonValue"), - ("public", "json_value", "other_json_value", "OtherJsonValue"), - ], - ['Duplicate custom type name mapping source "public.json_value"'], - ), - ( - [("public", "missing", "pg_missing", "PgMissing")], - ['Unknown custom type name mapping source "public.missing"'], - ), - ( - [("public", "json_value", "pg_json_value", "NoRowError")], - [ - "Generated Python name collision in package facade", - "generated core symbol NoRowError", - 'custom type "public.json_value"', - 'both resolve to "NoRowError"', - ], - ), - ( - [("public", "json_value", "pg_json_value", "JsonValue")], - [ - "Generated Python name collision in package facade", - "generated core symbol JsonValue", - 'custom type "public.json_value"', - 'both resolve to "JsonValue"', - ], - ), - ( - [("public", "json_value", "pg_json_value", "pgJsonValue")], - ["Mapped custom type pascalCase must start with an uppercase ASCII letter"], - ), - ( - [("public", "json_value", "__path__", "PgJsonValue")], - ["Mapped custom type snakeCase must start with a lowercase ASCII letter"], - ), - ], - ids=[ - "duplicate-source", - "unknown-source", - "mapped-final-collision", - "mapped-json-value", - "lowercase-pascal", - "path-module", - ], -) -def test_invalid_custom_type_mappings_fail_before_emission( - mappings: Sequence[CustomTypeMapping], - expected: Sequence[str], - pgn_bin: str, - pgn_admin_url: str, - tmp_path: Path, -) -> None: - project = _fresh_project( - tmp_path, - migration="CREATE TYPE \"json_value\" AS ENUM ('ready');\n", - queries={"read_json_value.sql": "SELECT 'ready'::public.\"json_value\" AS value\n"}, - custom_type_mappings=mappings, - ) - _analyse(pgn_bin, pgn_admin_url, project) - _assert_generation_fails(pgn_bin, pgn_admin_url, project, expected) diff --git a/tests/test_unsupported_types.py b/tests/test_unsupported_types.py index e9fd292..2625c3e 100644 --- a/tests/test_unsupported_types.py +++ b/tests/test_unsupported_types.py @@ -122,11 +122,6 @@ def _write_contract_probe( dimensionality: int, nested: bool, ) -> None: - array_settings = ( - "None Model.ArraySettings" - if dimensionality == 0 - else f"Some {{ dimensionality = {dimensionality}, elementIsNullable = False }}" - ) lookup = { "absent": "CustomKind.TypeKind.Absent", "enum": ('CustomKind.TypeKind.Enum { className = "ProbeValue", moduleName = "probe_value", order = 0 }'), @@ -134,9 +129,6 @@ def _write_contract_probe( 'CustomKind.TypeKind.Composite { className = "ProbeValue", moduleName = "probe_value", order = 0 }' ), }[lookup_kind] - interpreter_config = ( - "{ customTypeNameMappings = [] : List PythonNameMapping.CustomType }" if interpreter == "CustomType" else "{=}" - ) target_input = "customType" if nested else "member" custom_type = ( """ @@ -166,8 +158,6 @@ def _write_contract_probe( let CustomKind = ./Structures/CustomKind.dhall -let PythonNameMapping = ./Structures/PythonNameMapping.dhall - let Target = ./Interpreters/{interpreter}.dhall let Config = @@ -182,8 +172,7 @@ def _write_contract_probe( , onUnsupported = None OnUnsupported.Mode }} -let interpreterConfig = - {interpreter_config} +let interpreterConfig = {{=}} let run = \\(_ : Config) -> @@ -196,8 +185,15 @@ def _write_contract_probe( , name , pgName = "probe_value" , value = - {{ arraySettings = {array_settings} - , scalar = Model.Scalar.Custom name + {{ dimensionality = {dimensionality} + , elementIsNullable = False + , scalar = + Model.Scalar.Custom + {{ name + , pgSchema = "public" + , pgName = "probe_value" + , index = 0 + }} }} }} @@ -382,7 +378,7 @@ def test_custom_shape_contracts_succeed( assert result.returncode == 0, f"expected {case_id} to succeed:\n{_combined_output(result)}" -def test_skip_preserves_mapped_nested_registration_chain( +def test_skip_preserves_nested_registration_chain( pgn_bin: str, pgn_admin_url: str, tmp_path: Path, @@ -402,22 +398,11 @@ def test_skip_preserves_mapped_nested_registration_chain( "SELECT ROW(ROW('ready'::z_survivor_status)::m_survivor_leaf)::n_survivor_outer AS value\n" ) _ = (project / "queries" / "read_bad.sql").write_text("SELECT ROW(1::money)::a_bad AS value\n") - _write_single_artifact(project, "../../src/package.dhall", "skip-mapping-chain", "Skip") - config = project / "project1.pgn.yaml" - _ = config.write_text( - config.read_text() - + " customTypeNameMappings:\n" - + " - source:\n" - + " schema: public\n" - + " name: z_survivor_status\n" - + " target:\n" - + " snakeCase: mapped_status\n" - + " pascalCase: MappedStatus\n" - ) + _write_single_artifact(project, "../../src/package.dhall", "skip-nested-chain", "Skip") result = run_pgn(pgn_bin, pgn_admin_url, project, "generate") diagnostic = re.sub(r"\x1b\[[0-9;?]*[A-Za-z]", "", _combined_output(result)) - assert result.returncode == 0, f"mapped Skip generation failed:\n{diagnostic}" + assert result.returncode == 0, f"Skip generation failed:\n{diagnostic}" warnings = re.findall(r"Warning: ([^\n]+)\nStage: ([^\n]+)", diagnostic) assert warnings == [ ("Unsupported type", "Generating > python > Compiling > money > amount > a_bad"), @@ -427,13 +412,13 @@ def test_skip_preserves_mapped_nested_registration_chain( ), ] - package = project / "artifacts" / "python" / "src" / "skip_mapping_chain" + package = project / "artifacts" / "python" / "src" / "skip_nested_chain" generated = package / "_generated" types = generated / "types" assert {path.name for path in types.glob("*.py")} == { "__init__.py", "m_survivor_leaf.py", - "mapped_status.py", + "z_survivor_status.py", "n_survivor_outer.py", } @@ -445,9 +430,9 @@ def test_skip_preserves_mapped_nested_registration_chain( leaf_source = (types / "m_survivor_leaf.py").read_text() outer_source = (types / "n_survivor_outer.py").read_text() assert [line for line in leaf_source.splitlines() if line.startswith("from .")] == [ - "from .mapped_status import MappedStatus" + "from .z_survivor_status import ZSurvivorStatus" ] - assert "status: MappedStatus" in leaf_source + assert "status: ZSurvivorStatus" in leaf_source assert [line for line in outer_source.splitlines() if line.startswith("from .")] == [ "from .m_survivor_leaf import MSurvivorLeaf" ] @@ -458,7 +443,7 @@ def test_skip_preserves_mapped_nested_registration_chain( line for line in register_source.splitlines() if line.startswith("_") and "_pg_name = " in line ] assert pg_name_constants == [ - '_mapped_status_pg_name = "public.z_survivor_status"', + '_z_survivor_status_pg_name = "public.z_survivor_status"', '_m_survivor_leaf_pg_name = "public.m_survivor_leaf"', '_n_survivor_outer_pg_name = "public.n_survivor_outer"', ] From 9856f2a7f3396648df051467b4cacb34bc7578b1 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Wed, 15 Jul 2026 00:31:21 +0300 Subject: [PATCH 10/14] Replace buildLookup's Text/equal cascade with gen-sdk CustomTypes Retire the hand-rolled buildLookup custom-kind resolver (the last Text/equal user in src/) in favor of gen-sdk v3's Sdk.CustomTypes. References now resolve by gen-contract v5's CustomTypeRef.index via CustomKind.at instead of a Model.Name -> TypeKind closure keyed on Text/equal. - CustomKind.Lookup is now a List TypeKind indexed by CustomTypeRef.index, with an `at` accessor (out-of-range reads as Absent). - Member/ParamsMember/CustomType resolve refs by index; CustomType.run gains an explicit self-index parameter for its self-lookup consistency check. - Project.dhall builds a fixed, never-shrinking kind classification (kindOf) and, in Skip mode, a one-pass survivor cascade via supportedCustomTypesReasoned, then masks unsupported indices to Absent so a reference to a removed type reads exactly as a missing type (Absent on either failure). Fail mode masks nothing, so its behavior is unchanged. - The survivor predicate (ownDefinitionSupported) mirrors CustomTypeGen.run's own-shape failure modes exactly: Domain unsupported, Enum always supported, Composite supported iff every primitive member is a supported primitive (oracle: Primitive.run) and every custom member is a bare scalar. This is stricter than a kind-based array-rank check and is required so no surviving type later fails to render under Skip (exercised by test_skip_preserves_nested_registration_chain and test_skip_unsupported_drops_offending_units_and_cascades). - Update the contract-probe test helper for the List-shaped lookup and the new CustomType self-index argument. - Refresh tests/golden: pgn v0.12.0 now emits Project.customTypes in the gen-contract v5 topological order (dependencies first), reordering only the facade/types-init export lists (a_codec_wrapper after z_codec_payload). No generated type body or statement changes; the generator preserves input order. --- src/Interpreters/CustomType.dhall | 9 +- src/Interpreters/Member.dhall | 2 +- src/Interpreters/ParamsMember.dhall | 2 +- src/Interpreters/Project.dhall | 269 ++++++++++++------ src/Structures/CustomKind.dhall | 22 +- tests/golden/src/specimen_client/__init__.py | 8 +- .../_generated/types/__init__.py | 2 +- .../src/specimen_client/sync/__init__.py | 8 +- tests/test_unsupported_types.py | 11 +- 9 files changed, 227 insertions(+), 106 deletions(-) diff --git a/src/Interpreters/CustomType.dhall b/src/Interpreters/CustomType.dhall index cdd259b..b158fce 100644 --- a/src/Interpreters/CustomType.dhall +++ b/src/Interpreters/CustomType.dhall @@ -73,6 +73,11 @@ let renderExtraImports = let run = \(config : Config) -> \(lookup : CustomKind.Lookup) -> + -- The custom type's own position in the project's `customTypes` list. + -- Unlike a `CustomTypeRef`, a `Model.CustomType` does not carry its own + -- index, so the caller (Interpreters/Project.dhall) threads it in for the + -- self-lookup kind-consistency check in each merge arm below. + \(index : Natural) -> \(input : Input) -> let pythonName = { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase @@ -130,7 +135,7 @@ let run = [ input.pgName ] "Custom type not found in project customTypes" } - (lookup input.name) + (CustomKind.at lookup index) , Composite = \(members : List Model.Member) -> let compiledMembers @@ -224,7 +229,7 @@ let run = [ input.pgName ] "Custom type not found in project customTypes" } - (lookup input.name) + (CustomKind.at lookup index) in Lude.Compiled.flatMap (List MemberGen.Output) diff --git a/src/Interpreters/Member.dhall b/src/Interpreters/Member.dhall index bbe05b3..bffb7bb 100644 --- a/src/Interpreters/Member.dhall +++ b/src/Interpreters/Member.dhall @@ -113,7 +113,7 @@ let runWithPrefix = [ ref.name.inSnakeCase ] "Custom type not found in project customTypes" } - (lookup ref.name) + (CustomKind.at lookup ref.index) ) ( Lude.Compiled.report Output diff --git a/src/Interpreters/ParamsMember.dhall b/src/Interpreters/ParamsMember.dhall index bf42326..d2fb6d3 100644 --- a/src/Interpreters/ParamsMember.dhall +++ b/src/Interpreters/ParamsMember.dhall @@ -320,7 +320,7 @@ let run = [ ref.name.inSnakeCase ] "Custom type not found in project customTypes" } - (lookup ref.name) + (CustomKind.at lookup ref.index) ) ( if isJsonArrayParam then Lude.Compiled.report diff --git a/src/Interpreters/Project.dhall b/src/Interpreters/Project.dhall index e7d06fb..ddee955 100644 --- a/src/Interpreters/Project.dhall +++ b/src/Interpreters/Project.dhall @@ -14,6 +14,8 @@ let QueryGen = ./Query.dhall let CustomTypeGen = ./CustomType.dhall +let PrimitiveGen = ./Primitive.dhall + let CoreModule = ../Templates/CoreModule.dhall let RuntimeModule = ../Templates/RuntimeModule.dhall @@ -120,36 +122,92 @@ let resolveCustomTypes = ) (Prelude.List.indexed Model.CustomType customTypes) -let lookupEntries = - \(customTypes : List ResolvedCustomType) -> +-- The fixed, never-shrinking kind classification of every custom type, in +-- `input.customTypes` order (so position i == a `CustomTypeRef.index` of i). +-- A type's kind (Enum/Composite/Domain) is a property of its own definition +-- and never changes across Skip removal passes; Domain has no Python identity, +-- so it classifies as `Absent`. Survivorship is layered on separately (see +-- `supported`/`lookup` in `run`), keeping the two concerns that the former +-- `buildLookup` conflated cleanly apart. +let kindOf + : List ResolvedCustomType -> CustomKind.Lookup + = \(entries : List ResolvedCustomType) -> Prelude.List.map ResolvedCustomType - LookupEntry - (\(customType : ResolvedCustomType) -> customType.lookupEntry) - customTypes - -let buildLookup = - \(entries : List LookupEntry) -> - Prelude.List.fold - LookupEntry + CustomKind.TypeKind + ( \(rt : ResolvedCustomType) -> + merge + { Composite = + CustomKind.TypeKind.Composite rt.lookupEntry.identity + , Enum = CustomKind.TypeKind.Enum rt.lookupEntry.identity + , Domain = CustomKind.TypeKind.Absent + } + rt.lookupEntry.kind + ) entries - CustomKind.Lookup - ( \(entry : LookupEntry) -> - \(rest : CustomKind.Lookup) -> - \(name : Model.Name) -> - -- Text/equal is a pgn embedded-Dhall builtin. The upstream exit is - -- a stable custom kind or id on Scalar.Custom. - if Text/equal name.inSnakeCase entry.contractName - then merge - { Composite = - CustomKind.TypeKind.Composite entry.identity - , Enum = CustomKind.TypeKind.Enum entry.identity - , Domain = CustomKind.TypeKind.Absent + +-- True when the primitive has a faithful Python mapping (`money`, `bit`, geo +-- types, etc. do not). Reuses the Primitive interpreter as the oracle rather +-- than duplicating its 60-way supported/unsupported table, so this predicate +-- can never drift from what `CustomTypeGen.run` actually emits. +let primitiveSupported = + \(primitive : Model.Primitive) -> + merge + { Ok = + \(_ : { warnings : List Report, value : PrimitiveGen.Output }) -> + True + , Err = \(_ : Report) -> False + } + (PrimitiveGen.run {=} primitive) + +-- Whether a custom type's OWN definition compiles, assuming every custom type +-- it references is present (the transitive "is a referenced type still a +-- survivor" half is handled by `Sdk.CustomTypes.supportedCustomTypesReasoned`, +-- which folds this predicate over the topologically-sorted `customTypes`). +-- This exactly mirrors the own-shape failure modes of `CustomTypeGen.run`: +-- * Domain -> never supported (no Python lowering). +-- * Enum -> always compiles. +-- * Composite -> every member must compile: a primitive member needs a +-- supported primitive; a custom member must be a bare +-- scalar (CustomType.dhall rejects any custom array +-- field, regardless of the referenced kind). +-- Using this as the survivorship predicate (instead of the former +-- `buildLookup`-based render probe) keeps Skip mode dropping exactly the set of +-- types that could not be emitted, so no surviving type later fails to render. +let ownDefinitionSupported + : Model.CustomTypeDefinition -> Bool + = \(definition : Model.CustomTypeDefinition) -> + merge + { Composite = + \(members : List Model.Member) -> + Prelude.List.all + Model.Member + ( \(member : Model.Member) -> + merge + { Primitive = + \(primitive : Model.Primitive) -> + primitiveSupported primitive + , Custom = + \(_ : Model.CustomTypeRef) -> + Natural/isZero member.value.dimensionality } - entry.kind - else rest name - ) - (\(_ : Model.Name) -> CustomKind.TypeKind.Absent) + member.value.scalar + ) + members + , Enum = \(_ : List Model.EnumVariant) -> True + , Domain = \(_ : Model.Value) -> False + } + definition + +let boolAt = + \(bools : List Bool) -> + \(index : Natural) -> + Prelude.Optional.fold + Bool + (Prelude.List.index index Bool bools) + Bool + (\(b : Bool) -> b) + False let sameOrder = \(left : Natural) -> @@ -499,78 +557,90 @@ let run = let skip = merge { Fail = False, Skip = True } resolvedConfig.onUnsupported - let typeSucceedsWith = - \(candidateLookup : CustomKind.Lookup) -> - \(ct : Model.CustomType) -> - merge - { Ok = - \(_ : { value : CustomTypeGen.Output, warnings : List Report }) -> - True - , Err = \(_ : Report) -> False - } - (CustomTypeGen.run customTypeConfig candidateLookup ct) + let resolvedCustomTypes = resolveCustomTypes input.customTypes - -- Nested under the type's own name so the warning names the type - -- that failed, not just the inner member/column that triggered it - -- (CustomType.run itself does not). - let typeWarningWith = - \(candidateLookup : CustomKind.Lookup) -> - \(ct : Model.CustomType) -> - merge - { Ok = - \(_ : { value : CustomTypeGen.Output, warnings : List Report }) -> - None Report - , Err = - \(err : Report) -> Some { path = [ ct.name.inSnakeCase ] # err.path, message = err.message } - } - (CustomTypeGen.run customTypeConfig candidateLookup ct) + -- The unmasked, index-aligned kind classification of every custom type. + let fixedKindOf = kindOf resolvedCustomTypes + + -- Per-index survivorship. In Skip mode this is the one-pass transitive + -- cascade from gen-sdk: `supportedCustomTypesReasoned` folds + -- `ownDefinitionSupported` over the topologically-sorted `customTypes`, + -- returning `None` (supported) or `Some ` (unsupported) + -- per index; we keep only the Bool. In Fail mode nothing is ever + -- dropped, so every index is supported. + let supported + : List Bool + = if skip + then Prelude.List.map + (Optional Model.CustomTypeRef) + Bool + (Prelude.Optional.null Model.CustomTypeRef) + ( Sdk.CustomTypes.supportedCustomTypesReasoned + ownDefinitionSupported + input.customTypes + ) + else Prelude.List.map + ResolvedCustomType + Bool + (\(_ : ResolvedCustomType) -> True) + resolvedCustomTypes - let resolvedCustomTypes = resolveCustomTypes input.customTypes + -- The kind lookup handed to every interpreter. In Skip mode an + -- unsupported index is masked to `Absent` regardless of its real kind, + -- so a reference to a removed type reads exactly as a missing type + -- (the "Absent on either failure" invariant the former `buildLookup` + -- provided by shrinking its entry list). In Fail mode it is the + -- unmasked classification, so behaviour is identical to before Skip. + let lookup + : CustomKind.Lookup + = if skip + then Prelude.List.map + { index : Natural, value : CustomKind.TypeKind } + CustomKind.TypeKind + ( \(e : { index : Natural, value : CustomKind.TypeKind }) -> + if boolAt supported e.index + then e.value + else CustomKind.TypeKind.Absent + ) + (Prelude.List.indexed CustomKind.TypeKind fixedKindOf) + else fixedKindOf + + let indexedResolvedCustomTypes + : List { index : Natural, value : ResolvedCustomType } + = Prelude.List.indexed ResolvedCustomType resolvedCustomTypes - -- Nested custom support requires transitive closure: after one type is - -- removed, composites depending on it must be reconsidered against the - -- smaller lookup. Each bounded pass can only remove survivors. let effectiveResolvedCustomTypes - : List ResolvedCustomType + : List { index : Natural, value : ResolvedCustomType } = if skip - then Natural/fold - (Prelude.List.length Model.CustomType input.customTypes) - (List ResolvedCustomType) - ( \(survivors : List ResolvedCustomType) -> - let candidateLookup = - buildLookup (lookupEntries survivors) - - in Prelude.List.filter - ResolvedCustomType - ( \(customType : ResolvedCustomType) -> - typeSucceedsWith - candidateLookup - customType.value - ) - survivors + then Prelude.List.filter + { index : Natural, value : ResolvedCustomType } + ( \(e : { index : Natural, value : ResolvedCustomType }) -> + boolAt supported e.index ) - resolvedCustomTypes - else resolvedCustomTypes + indexedResolvedCustomTypes + else indexedResolvedCustomTypes - let effectiveCustomTypes = - Prelude.List.map - ResolvedCustomType - Model.CustomType - (\(customType : ResolvedCustomType) -> customType.value) + let effectiveCustomTypes + : List { index : Natural, value : Model.CustomType } + = Prelude.List.map + { index : Natural, value : ResolvedCustomType } + { index : Natural, value : Model.CustomType } + ( \(e : { index : Natural, value : ResolvedCustomType }) -> + { index = e.index, value = e.value.value } + ) effectiveResolvedCustomTypes - let lookup = - buildLookup (lookupEntries effectiveResolvedCustomTypes) - - -- Fail mode: identical to the pre-Skip code (traverseList straight - -- over input.customTypes), so its error message/path is unchanged. + -- Fail mode: `effectiveCustomTypes` is every input type (indexed), so + -- this is the pre-Skip traversal with each type's own index threaded to + -- CustomTypeGen.run for its self-lookup; its error message/path is + -- unchanged. Skip mode traverses only the survivors. let typesForCombine : Lude.Compiled.Type (List CustomTypeGen.Output) = Lude.Compiled.traverseList - Model.CustomType + { index : Natural, value : Model.CustomType } CustomTypeGen.Output - ( \(ct : Model.CustomType) -> - CustomTypeGen.run customTypeConfig lookup ct + ( \(e : { index : Natural, value : Model.CustomType }) -> + CustomTypeGen.run customTypeConfig lookup e.index e.value ) effectiveCustomTypes @@ -620,16 +690,39 @@ let run = registrationOrder typesForCombine + -- Nested under the type's own name so the warning names the type that + -- failed, not just the inner member/column that triggered it + -- (CustomType.run itself does not). Uses the single final `lookup` + -- (which already masks removed types to Absent) and each type's own + -- real index, so every input type is diagnosed at its true position. + let typeWarning = + \(ct : Model.CustomType) -> + \(index : Natural) -> + merge + { Ok = + \(_ : { value : CustomTypeGen.Output, warnings : List Report }) -> + None Report + , Err = + \(err : Report) -> + Some + { path = [ ct.name.inSnakeCase ] # err.path + , message = err.message + } + } + (CustomTypeGen.run customTypeConfig lookup index ct) + let skipWarnings : List Report = if skip then Prelude.List.unpackOptionals Report ( Prelude.List.map - Model.CustomType + { index : Natural, value : ResolvedCustomType } (Optional Report) - (typeWarningWith lookup) - input.customTypes + ( \(e : { index : Natural, value : ResolvedCustomType }) -> + typeWarning e.value.value e.index + ) + indexedResolvedCustomTypes ) # Prelude.List.unpackOptionals Report diff --git a/src/Structures/CustomKind.dhall b/src/Structures/CustomKind.dhall index 1621f13..4748022 100644 --- a/src/Structures/CustomKind.dhall +++ b/src/Structures/CustomKind.dhall @@ -1,4 +1,4 @@ -let Model = ../Deps/Contract.dhall +let Prelude = ../Deps/Prelude.dhall let Identity = { className : Text, moduleName : Text, order : Natural } @@ -9,6 +9,22 @@ let TypeKind = | Absent > -let Lookup = Model.Name -> TypeKind +-- Index-aligned with a project's `customTypes` list: `Lookup` at position `i` +-- is the kind classification of `customTypes[i]`. Replaces the former +-- `Model.Name -> TypeKind` closure (which relied on `Text/equal`), so a +-- reference resolves by its `CustomTypeRef.index` instead of by name. +let Lookup = List TypeKind -in { TypeKind, Lookup, Identity } +-- Resolve a reference's kind by index; an out-of-range index reads as `Absent` +-- (mirrors gen-sdk's own `optionalIndex` helper in supportedCustomTypes.dhall). +let at = + \(lookup : Lookup) -> + \(index : Natural) -> + Prelude.Optional.fold + TypeKind + (Prelude.List.index index TypeKind lookup) + TypeKind + (\(kind : TypeKind) -> kind) + TypeKind.Absent + +in { TypeKind, Lookup, Identity, at } diff --git a/tests/golden/src/specimen_client/__init__.py b/tests/golden/src/specimen_client/__init__.py index 7f11f41..a99ea57 100644 --- a/tests/golden/src/specimen_client/__init__.py +++ b/tests/golden/src/specimen_client/__init__.py @@ -82,9 +82,6 @@ # isort: on # isort: off -from ._generated.types.a_codec_wrapper import ( - ACodecWrapper as ACodecWrapper, -) from ._generated.types.mood import ( Mood as Mood, ) @@ -97,6 +94,9 @@ from ._generated.types.z_codec_payload import ( ZCodecPayload as ZCodecPayload, ) +from ._generated.types.a_codec_wrapper import ( + ACodecWrapper as ACodecWrapper, +) # isort: on __all__ = [ @@ -105,11 +105,11 @@ ] # Project order is intentional for public re-export groups. __all__ += [ # noqa: RUF022, RUF100 - "ACodecWrapper", "Mood", "Point2D", "TagValue", "ZCodecPayload", + "ACodecWrapper", ] __all__ += [ # noqa: RUF022, RUF100 "GetSpecimenRow", diff --git a/tests/golden/src/specimen_client/_generated/types/__init__.py b/tests/golden/src/specimen_client/_generated/types/__init__.py index 9a8d597..3b56de9 100644 --- a/tests/golden/src/specimen_client/_generated/types/__init__.py +++ b/tests/golden/src/specimen_client/_generated/types/__init__.py @@ -3,9 +3,9 @@ # SPDX-License-Identifier: MIT-0 # isort: off -from .a_codec_wrapper import ACodecWrapper as ACodecWrapper from .mood import Mood as Mood from .point_2_d import Point2D as Point2D from .tag_value import TagValue as TagValue from .z_codec_payload import ZCodecPayload as ZCodecPayload +from .a_codec_wrapper import ACodecWrapper as ACodecWrapper # isort: on diff --git a/tests/golden/src/specimen_client/sync/__init__.py b/tests/golden/src/specimen_client/sync/__init__.py index 4a89f76..21d00e8 100644 --- a/tests/golden/src/specimen_client/sync/__init__.py +++ b/tests/golden/src/specimen_client/sync/__init__.py @@ -79,9 +79,6 @@ # isort: on # isort: off -from .._generated.types.a_codec_wrapper import ( - ACodecWrapper as ACodecWrapper, -) from .._generated.types.mood import ( Mood as Mood, ) @@ -94,6 +91,9 @@ from .._generated.types.z_codec_payload import ( ZCodecPayload as ZCodecPayload, ) +from .._generated.types.a_codec_wrapper import ( + ACodecWrapper as ACodecWrapper, +) # isort: on __all__ = [ @@ -102,11 +102,11 @@ ] # Project order is intentional for public re-export groups. __all__ += [ # noqa: RUF022, RUF100 - "ACodecWrapper", "Mood", "Point2D", "TagValue", "ZCodecPayload", + "ACodecWrapper", ] __all__ += [ # noqa: RUF022, RUF100 "GetSpecimenRow", diff --git a/tests/test_unsupported_types.py b/tests/test_unsupported_types.py index 2625c3e..b88cfb4 100644 --- a/tests/test_unsupported_types.py +++ b/tests/test_unsupported_types.py @@ -130,6 +130,13 @@ def _write_contract_probe( ), }[lookup_kind] target_input = "customType" if nested else "member" + # CustomKind.Lookup is now a `List CustomKind.TypeKind` indexed by + # CustomTypeRef.index; these probes reference exactly one custom type + # ("ProbeValue") at index 0, so a one-element list suffices. CustomType.run + # also gained an explicit self-index parameter (0 here), threaded only for + # the CustomType interpreter (Member/ParamsMember resolve by ref.index and + # never needed a self-index). + index_argument = " 0" if interpreter == "CustomType" else "" custom_type = ( """ let customType @@ -200,13 +207,13 @@ def _write_contract_probe( {custom_type} let lookup : CustomKind.Lookup - = \\(_ : Model.Name) -> {lookup} + = [ {lookup} ] in Lude.Compiled.map Target.Output Lude.Files.Type (\\(_ : Target.Output) -> [] : Lude.Files.Type) - (Target.run interpreterConfig lookup {target_input}) + (Target.run interpreterConfig lookup{index_argument} {target_input}) in Sdk.Sigs.generator Config Config/default run """ From 7240744e302a4a6bb3ede4f3793d2a72e7cc5a41 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Wed, 15 Jul 2026 00:31:31 +0300 Subject: [PATCH 11/14] Docs: buildLookup replaced by gen-sdk CustomTypes; no Text/equal left in src - README/DESIGN: describe the index-based reference resolution and the single-pass Sdk.CustomTypes survivor cascade; record that no live Text/equal use remains anywhere in src/ (buildLookup was the last one). - CHANGELOG (Upcoming): add the v5.0.0/v3.0.0 pin bump + buildLookup retirement bullet; reconcile the now-superseded "Retained buildLookup" and "bounded fixed point" wording in the same unreleased section. - upstream-asks: mark the qualified-reference ask as delivered by gen-contract v5 and correct the stale buildLookup current-mechanism claim. --- CHANGELOG.md | 34 ++++++++++++++---------- DESIGN.md | 61 ++++++++++++++++++++++++------------------- README.md | 11 +++++--- docs/upstream-asks.md | 21 ++++++++------- 4 files changed, 73 insertions(+), 54 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9af5c9..79a096e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Upcoming +- Bumped the pins to gen-contract v5.0.0 and gen-sdk v3.0.0 (requires `pgn` + v0.12.0). Replaced the hand-rolled `buildLookup` custom-kind resolver and its + fixed-point removal cascade with gen-sdk's `CustomTypes` module: references + now resolve by the contract's `CustomTypeRef.index` and `onUnsupported: Skip` + computes survivorship in a single left-fold over the topologically-sorted + `customTypes`. This retires the generator's last `Text/equal` use, so no live + `Text/equal` call remains anywhere in `src/`. + - Removed the `queryNameMappings`/`customTypeNameMappings` rename-mapping config and all generation-time Python namespace-collision detection (project-wide facade/module, per-query, per-custom-type, and per-type @@ -33,22 +41,20 @@ ranks, composite ranks, custom-array fields inside composites, and missing or unsupported custom types fail loudly. -- Retained `buildLookup` as the project-wide custom-kind resolver for identities - preserved in the contract. Its `Text/equal` use is an explicit pgn-fork - constraint. The planned exit must preserve every schema-qualified custom type - entry and expose a stable qualified identifier on `Scalar.Custom`; kind alone - is insufficient. pgn 0.9.1 can collapse same-unqualified-name types across - schemas, so that database shape remains unsupported and cannot be repaired by - a Python mapping. Natural project indexes now provide deterministic - custom-import deduplication and ordering. - Query, parameter, field, and private statement names are collision-safe while - SQL names remain unchanged. +- Resolved custom-type references by their contract-supplied + `CustomTypeRef.index` (gen-contract v5) rather than by name comparison, so a + reference carries a stable schema-qualified identity (`pgSchema`/`pgName`) + directly. Same-unqualified-name types across schemas therefore no longer + collapse. Natural project indexes provide deterministic custom-import + deduplication and ordering. Query, parameter, field, and private statement + names are collision-safe while SQL names remain unchanged. - Kept `onUnsupported: Fail | Skip`. `Fail` aborts generation with the nested - report. `Skip` preserves warnings and computes a bounded fixed point that - removes an unsupported custom type, its dependent custom types and statements, - and all affected type, registration, facade, Row, and statement entries. The - surviving package remains strict-importable. + report. `Skip` preserves warnings and computes survivorship in a single + left-fold over the topologically-sorted `customTypes`, removing an unsupported + custom type, its dependent custom types and statements, and all affected type, + registration, facade, Row, and statement entries. The surviving package + remains strict-importable. - Established the generated tree as a greenfield layout with no compatibility layer or promise for internal generated paths. Every generated Python file now diff --git a/DESIGN.md b/DESIGN.md index ab7e825..c33ec31 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -257,25 +257,31 @@ postprocessor rewrites generated files. ```text optional config -> resolved config - -> buildLookup(custom types) - -> fixed-point custom-type filtering in Skip mode - -> compile custom types and query checks + -> index-aligned kind classification (kindOf) of every custom type + -> one-pass survivor cascade (Sdk.CustomTypes.supportedCustomTypesReasoned) + masking unsupported indices to Absent in Skip mode + -> compile custom types and query checks against the masked lookup -> dependency-first registration order -> render facades, core, runtimes, types, registration, and statements -> prepend the generated header ``` `onUnsupported: Fail` traverses the original project and propagates the first -compiled failure, so generation aborts without a partial client. - -`onUnsupported: Skip` repeatedly rebuilds `buildLookup` from the surviving -custom types and removes any type that no longer compiles. It then compiles -queries against the final lookup. A removed type therefore also removes its -dependent composites and statements. Facade exports, type initializers, -registration entries, statement files, and Row exports are assembled only from -survivors. Reports for dropped units are preserved as warnings. The bounded -fixed point can only remove candidates, so it terminates after at most the -original custom-type count. +compiled failure, so generation aborts without a partial client. In this mode +nothing is masked: the lookup is the full, unmasked kind classification. + +`onUnsupported: Skip` computes survivorship in a single left-fold via +`Sdk.CustomTypes.supportedCustomTypesReasoned`. Because `Project.customTypes` +is topologically sorted (every referenced index precedes the referencing type), +one pass suffices: the fold marks a type unsupported if its own definition +cannot compile (`ownDefinitionSupported`) or if any custom type it references +was already marked unsupported. Unsupported indices are then masked to `Absent` +in the kind lookup, so a reference to a removed type reads exactly like a +missing type. Queries compile against that same masked lookup, so a removed type +also removes its dependent composites and statements. Facade exports, type +initializers, registration entries, statement files, and Row exports are +assembled only from survivors, and reports for dropped units are preserved as +warnings. Query compilation is deliberately consolidated in `QueryCheck`: keep/drop state and warning data come from one literal `QueryGen.run` call site before the final @@ -350,20 +356,21 @@ to guarantee unique custom-type identities at the source instead. The generated package's `basedpyright --strict` gate (see section 12) is now the only backstop against a Python name collision reaching a consumer. -`buildLookup` intentionally remains in `Interpreters/Project.dhall`. It compares -a custom reference's snake-case name with the project custom type name using -`Text/equal`. That builtin belongs to pgn's embedded Dhall fork and is not -available in upstream standard Dhall. It is, as of this writing, the only -remaining `Text/equal` use anywhere in `src/` — a repo-wide grep confirms it. - -The dependency is pinned and explicit. The complete fixture needs fork-aware -evaluation solely because it invokes this generator and the local `buildLookup` -uses `Text/equal`. The upstream exit must preserve every schema-qualified -`customTypes` entry and put a stable qualified identifier, or a project index -with equivalent identity, on each custom scalar reference. Besides removing -text equality, that prevents pgn 0.9.1 from collapsing same-unqualified-name -types across schemas. Until then, pgn and CI's pinned fork-aware action are the -supported evaluators, and cross-schema duplicate type names are unsupported. +`buildLookup` — the last `Text/equal` user, which compared a reference's +snake-case name against each project custom type's name — has been removed. +gen-contract v5's `CustomTypeRef` carries an `index` into `Project.customTypes`, +and gen-sdk v3's `CustomTypes` module folds over that topologically-sorted list +to compute survivorship without any text comparison. References now resolve by +index (`Structures/CustomKind.dhall`'s `at`), so a repo-wide `grep -rn +"Text/equal" src/` finds only these explanatory comments — no live use remains. + +The generator no longer needs fork-only text equality anywhere. It does still +rely on gen-contract v5's contract guarantees: every `CustomTypeRef.index` must +address the intended `customTypes` entry, and `customTypes` must be +topologically sorted (every referenced index precedes the referencing type), +which is what makes the single-pass survivor cascade sound. `pgn` remains the +supported evaluator and generation driver; upholding those index/ordering +invariants is the producer's responsibility. `PyIdent.dhall` uses `Lude.Text.replaceIfOneOf`'s bounded `Text/replace` construction for keyword membership. `ImportSet.dhall` uses natural project diff --git a/README.md b/README.md index 6338461..43ce6d4 100644 --- a/README.md +++ b/README.md @@ -249,9 +249,14 @@ at a time, and rerun the harness. The release workflow resolves wheel from those exact bytes. The wheel exposes path, URL, and vendor commands; PyPI publication remains disabled until its explicit release gate is enabled. -`fixtures/Exhaustive.dhall` is the contract fixture. It and `buildLookup` rely -on the pgn fork's `Text/equal` builtin, so the standard upstream Dhall evaluator -cannot run the complete contract path. CI uses the pinned fork-aware action. +`fixtures/Exhaustive.dhall` is the contract fixture. The generator no longer +depends on the pgn fork's `Text/equal` builtin at all: the custom-type removal +cascade that was the last user (`buildLookup`) has been replaced by gen-sdk's +`CustomTypes` module, which resolves references by their contract-supplied +`CustomTypeRef.index` instead of by name comparison. `pgn` itself remains the +supported evaluator and generation driver (CI invokes `pgn generate`), and it +requires the contract's topological `customTypes` ordering that gen-contract +v5 guarantees. ## Provenance and license diff --git a/docs/upstream-asks.md b/docs/upstream-asks.md index 430fd50..d4d0c8a 100644 --- a/docs/upstream-asks.md +++ b/docs/upstream-asks.md @@ -32,16 +32,17 @@ codecs. The lookup classification instead drives support-shape validation, custom imports and dependencies, and dependency-first psycopg adapter registration. See `DESIGN.md`, sections 3, 5, and 8. -`Interpreters/Project.dhall` currently implements `buildLookup` by comparing a -custom reference's snake-case name with each project custom type through -`Text/equal`. This local lookup is the generator's sole need for that -pgn-specific builtin, but the unqualified comparison also exposes an upstream -identity gap. - -With pgn 0.9.1, a project containing `alpha.status` and `beta.status` can arrive -with only one `customTypes` entry, while both uses are represented by the same -unqualified `Scalar.Custom Name`. A Python mapping cannot recover the discarded -schema or safely repair annotations and adapter registration. +This ask has since been addressed by gen-contract v5.0.0 (see below). For +history: `Interpreters/Project.dhall` formerly implemented `buildLookup` by +comparing a custom reference's snake-case name with each project custom type +through `Text/equal`, which was the generator's sole need for that pgn-specific +builtin; the unqualified comparison also exposed an upstream identity gap. That +lookup has been removed — references now resolve by `CustomTypeRef.index`. + +Before gen-contract v5, a project containing `alpha.status` and `beta.status` +could arrive with only one `customTypes` entry, while both uses were represented +by the same unqualified `Scalar.Custom Name`. A Python mapping cannot recover a +discarded schema or safely repair annotations and adapter registration. The upstream ask has two inseparable parts: From b739344304101de53e84e3c72415b8a2eb1aca80 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Wed, 15 Jul 2026 01:43:59 +0300 Subject: [PATCH 12/14] Sweep stale doc status strings after final branch review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 4-task migration (gen-contract/gen-sdk bump, pgn v0.12.0 pin, Text/equal-dependent validation removal, buildLookup replacement) left several stale status strings that no single task owned: - README.md and DESIGN.md still cited the old pgn 0.9.1 pin and the old 73-passed test count; the suite is now 50 passed, 0 skipped after Task 3 deleted a 692-line test file for a discarded feature. - CHANGELOG.md's "Upcoming" section had two bullets contradicting each other on the pgn version and test count. - docs/upstream-asks.md repeated the stale pgn 0.9.1 pin statement. - DESIGN.md §9 and docs/upstream-asks.md §2 still described the old repeated-pass "fixed-point filtering" cascade that Task 4 replaced with Sdk.CustomTypes' single-pass left-fold; reworded both to match §8's already-correct description. - README.md's onUnsupported: Skip description also read as the old iterative "repeatedly removes... until fixed point" mechanism; reworded to describe the single pass over topologically-sorted custom types. Doc-only change; confirmed via mise run test (50 passed) and mise run lint. --- CHANGELOG.md | 4 ++-- DESIGN.md | 5 +++-- README.md | 12 ++++++------ docs/upstream-asks.md | 4 ++-- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 79a096e..e6420ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,8 +71,8 @@ class-aware adapters, pure models, canonical `args_row` Rows, and no query decoders, model codecs, casts, or bytes SQL. H2 CONFIRM: 25 Python files / 1467 lines / 11 statement files / 801 statement lines / 11 SQL. H3 CONFIRM: - Ruff 0/0, authored long0, SQL long0, raw output/no postformat. Tests: 73 - passed, 0 skipped; pgn 0.9.1; strict basedpyright 0/0. + Ruff 0/0, authored long0, SQL long0, raw output/no postformat. Tests: 50 + passed, 0 skipped; pgn v0.12.0; strict basedpyright 0/0. - Migrated the generator to gen-contract v4.0.1 and gen-sdk v2.0.0 using `Sdk.Sigs`. The implementation moved from `gen/` into `src/`, the working-tree diff --git a/DESIGN.md b/DESIGN.md index c33ec31..8788429 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -332,7 +332,8 @@ element nullability. `Member` and `ParamsMember` apply member nullability, identifier safety, lookup classification, imports, and parameter wrapping. `ResultColumns` builds Row field declarations in analyzed order. `Query` combines SQL, Row, parameters, and surfaces. `Project` owns lookup scope, -fixed-point filtering, registration order, file selection, and facades. +single-pass survivor filtering, registration order, file selection, and +facades. ## 10. The pinned `Text/equal` constraint @@ -405,7 +406,7 @@ PostgreSQL with these verdicts: - H2 CONFIRM: 25 Python files / 1467 lines / 11 statement files / 801 statement lines / 11 SQL. - H3 CONFIRM: Ruff 0/0, authored long0, SQL long0, raw output/no postformat. -- Tests: 73 passed, 0 skipped; pgn 0.9.1; strict basedpyright 0/0. +- Tests: 50 passed, 0 skipped; pgn v0.12.0; strict basedpyright 0/0. The H1 round trip covers both connection surfaces and exact cross-facade identities. It covers scalar enum, enum arrays including rank 2, scalar diff --git a/README.md b/README.md index 43ce6d4..bec88da 100644 --- a/README.md +++ b/README.md @@ -52,10 +52,10 @@ canonical statement module. `packageName` becomes an import name by replacing `-` with `_`. `onUnsupported: "Fail"` aborts generation with a path-aware report. -`onUnsupported: "Skip"` preserves warnings and repeatedly removes an -unsupported custom type, its dependent custom types and statements, and every -affected facade, registration, and statement entry until the survivors are a -strict-importable fixed point. +`onUnsupported: "Skip"` preserves warnings and, in a single pass over the +topologically-sorted custom types, removes an unsupported custom type, its +dependent custom types and statements, and every affected facade, +registration, and statement entry, leaving a strict-importable survivor set. Generated Python names are not audited for collisions at generation time. Query and custom-type identifiers are derived directly from their PostgreSQL @@ -219,7 +219,7 @@ disabled. ## Development -This repository pins pgn 0.9.1 in `mise.toml`. Run all tools through `mise`: +This repository pins pgn v0.12.0 in `mise.toml`. Run all tools through `mise`: ```bash mise run install @@ -229,7 +229,7 @@ mise run test The harness drives real pgn subprocesses against a live PostgreSQL server and only creates and drops its own scratch databases. Set `PGN_TEST_DATABASE_URL` -for a non-default server. The full verified suite is 73 passed and 0 skipped; +for a non-default server. The full verified suite is 50 passed and 0 skipped; it includes byte-for-byte golden comparison, async and sync round trips, basedpyright strict, Ruff, generated-quality budgets, unsupported-type modes, and public identity checks. diff --git a/docs/upstream-asks.md b/docs/upstream-asks.md index d4d0c8a..1755509 100644 --- a/docs/upstream-asks.md +++ b/docs/upstream-asks.md @@ -18,10 +18,10 @@ generator-side annotation parsing remains. Issue [#67](https://github.com/pgenie-io/pgenie/issues/67) is closed. Since pgn 0.7.2, successful generation surfaces reports from `Compiled.warnings`; this -repository pins pgn 0.9.1. +repository pins pgn v0.12.0. With `onUnsupported: Skip`, the generator retains a report for every dropped -unit while fixed-point filtering removes unsupported custom types, their +unit while a single-pass survivor fold removes unsupported custom types, their dependents, and affected statements. See `DESIGN.md`, section 8. ## 3. Preserve qualified custom-type identity: actionable From b64c505a6a4ca879629d29317e62e5c799ec3ac5 Mon Sep 17 00:00:00 2001 From: Nikita Volkov Date: Wed, 15 Jul 2026 03:27:39 +0300 Subject: [PATCH 13/14] Drop plans --- .../2026-07-14-detext-equal-branch-filter.md | 194 --- ...-07-14-gen-contract-v5-sdk-v3-migration.md | 1061 ----------------- 2 files changed, 1255 deletions(-) delete mode 100644 docs/plans/2026-07-14-detext-equal-branch-filter.md delete mode 100644 docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md diff --git a/docs/plans/2026-07-14-detext-equal-branch-filter.md b/docs/plans/2026-07-14-detext-equal-branch-filter.md deleted file mode 100644 index e05edfd..0000000 --- a/docs/plans/2026-07-14-detext-equal-branch-filter.md +++ /dev/null @@ -1,194 +0,0 @@ -# Handoff: filtering `fix-collision-stuff/2` down to what survives without `Text/equal` - -## Why this exists - -`fix-collision-stuff/2` accumulated 15 commits (all by Viacheslav Shvets) -since base commit `7b876a3e4f80d764ec16eff802288ec28ccda920`, several of -which lean on `Text/equal` — an experimental, pgn-fork-only Dhall builtin -that is going away in a future pgn/Dhall release. Stock Dhall has no text -equality primitive at all (a deliberate upstream omission), so any code -depending on `Text/equal` needs to be either eliminated or explicitly -accepted as blocked debt. This document is the outcome of a full audit of -which parts survive and which don't, and why. - -**Status when this was written:** decisions are locked (see table below). -Implementation has **not** started. Nikita was going to attempt a -gen-contract change first (see item 4) before any code gets written here, -since that changes what "keep, flagged as blocked" for `buildLookup` should -actually look like. - -**Update 2026-07-14 (later the same day):** the gen-contract change -happened and shipped — `gen-contract` v5.0.0 and `gen-sdk` v3.0.0 are now -released. Item 4 is no longer blocked debt; see its updated row and the -rewritten "What #4 needs" section below (now "What #4 gets"). This branch's -pins (`src/Deps/Contract.dhall` at v4.0.1, `src/Deps/Sdk.dhall` at v2.0.0) -have **not** been bumped yet — that bump plus the resulting migration is -now the actual next step, not a hypothetical. - -## Decision table - -| # | Item | Disposition | -|---|---|---| -| 1 | `emitSync` — dual sync+async generation in one module (`e46dc96`, `a1ac9f8`, `e7d1e6b`) | **Keep as-is.** Confirmed zero `Text/equal` dependency. | -| 2 | Custom type registration — psycopg-native adapter binding, dependency-ordered via a `Natural` `order` field (`bac8f95`, `375ba6f`) | **Keep as-is.** The registration/ordering mechanism itself is zero `Text/equal` (`sameOrder` uses `Natural/subtract`, not text comparison). | -| 3 | `50f74fa`'s module-internal reserved-name disambiguation (`querySafeName`, `moduleReservedNames`, `parameterSafeName` in `PyIdent.dhall`) | **Keep as-is.** Necessary regardless of any collision-detection decision (stops e.g. a query named `date` from shadowing its own generated file's `from datetime import date`), and already `Text/equal`-free via the same fixed-list `Text/replace` trick as Python-keyword escaping. | -| 4 | `buildLookup` (`5147204`, `Interpreters/Project.dhall:590-611`) — backs `onUnsupported: Skip`'s cascade removal and the empirically-verified rank-limit validation (composite arrays ≤1 dim, enum arrays ≤2 dims) | **Resolved, not yet implemented.** `gen-contract` v5.0.0 + `gen-sdk` v3.0.0 (released 2026-07-14) eliminate the need for `buildLookup`, `Text/equal`, and the hand-rolled `CustomKind.Lookup`/`Natural/fold` cascade entirely — see "What #4 gets" below. This branch's pins are still on v4.0.1/v2.0.0; bumping them and migrating `Interpreters/Project.dhall`/`Scalar.dhall`/`Structures/CustomKind.dhall` is now real work, not a keep-as-debt decision. | -| 5 | Flat top-level package facade (re-exports every query/type/core symbol into one shared top-level namespace) | **Replace.** Un-flatten into per-kind sub-namespaces (statements get their own namespace, types get their own). This is new work, not a keep/discard of an existing commit. | -| 6 | Same-kind collision *detection* (two queries, or two custom types, whose generated names collide with each other — as opposed to cross-kind collisions, which item 5 eliminates structurally) | **Discard.** Risk accepted; pushed upstream instead. Filed [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asking pgn to reject non-normalized query filenames and guarantee unique custom-type identities at the source. | -| 7 | Manual rename-override feature (`queryNameMappings`/`customTypeNameMappings` config, `Structures/PythonNameMapping.dhall`) | **Discard entirely.** Wasn't on the original keep-list; not needed once detection (item 6) is dropped; was the last thing in the collision cluster still needing `Text/equal` (to match a reference's name against a configured mapping's source). | -| 8 | `docs/adr/0001-generated-python-name-collisions.md` and related docs (`DESIGN.md`, `README.md`, `CHANGELOG.md` sections describing the flat facade / collision detection / rename mappings) | **Drop the ADR outright** (decided 2026-07-14, superseding the earlier "rewrite" plan). Related doc sections describing the now-discarded flat-facade/detection/rename-mapping behavior still need rewriting or removal to match the new state — not a blanket drop, just the ADR itself. | -| — | Tests exercising discarded behavior (`test_python_name_collisions.py`, `test_takeover_contract.py`) | Rewrite/remove to match — `test_python_name_collisions.py` specifically tests the collision-detection and rename-mapping mechanisms being discarded; `test_takeover_contract.py` locks in flat-facade output that item 5 changes. | - -## What #4 (buildLookup) gets from gen-contract v5.0.0 / gen-sdk v3.0.0 - -Both shipped 2026-07-14. What landed is better than the `id`/`kind` -addition originally drafted here: `gen-sdk` ships a ready-made, -`Text/equal`-free replacement for the whole cascade, not just the -identity tag. - -**gen-contract v5.0.0** (`Scalar.Custom` breaking change, commit -`e10128f`): the bare `Name` becomes a resolvable `CustomTypeRef`: - -```dhall -let CustomTypeRef = - { name : Name, pgSchema : Text, pgName : Text, index : Natural } - -let Scalar = < Primitive : Primitive | Custom : CustomTypeRef > -``` - -`index` points into `Project.customTypes : List CustomType`, and the -contract now guarantees that list is **topologically sorted**: every index -reachable from `customTypes[i]` is `< i`. So a reference resolves via -`Prelude.List.index ref.index _ project.customTypes` — a Natural-indexed -list lookup — never a name search, never `Text/equal`. Getting the -referenced type's *kind* (Composite/Enum/Domain) is then a plain union -`merge` on `.definition`, also `Text/equal`-free. - -**gen-sdk v3.0.0** (commit `bba067e` et al., pinned to gen-contract v5.0.0): -adds a `CustomTypes` module built on exactly one topological left fold over -`Project.customTypes`, exposing: - -- `supportedCustomTypes : (CustomTypeDefinition -> Bool) -> List CustomType -> List Bool` — - per-type supported/unsupported, propagated through nested - Composite/Domain dependencies via index lookups into the fold's own - running output (no re-traversal, no `Natural/fold`-over-`List/filter` - cascade like `effectiveResolvedCustomTypes` currently does). -- `supportedCustomTypesReasoned` — same fold, but returns the *root-cause* - `CustomTypeRef` for each unsupported type instead of a bare `Bool` - (useful for warning messages). -- `customTypeIsSupported : List Bool -> CustomTypeRef -> Bool` and - `queryIsSupported : List Bool -> Query -> Bool` — given the fold's output, - answer whether a specific reference or an entire query (all its params - and result columns, transitively) survives. - -This directly supersedes python.gen's own `buildLookup` + -`effectiveResolvedCustomTypes` cascade-removal loop -(`Interpreters/Project.dhall:1174-1204`), which was reinventing the same -topological propagation with repeated `Natural/fold` passes over -`Text/equal`-keyed lookups because the input wasn't guaranteed sorted -before. It does **not** replace the rank-limit checks in -`ParamsMember.dhall:294/318` and `Member.dhall:92/109` (array of enum >2 -dims / composite >1 dim) — those are python.gen-specific and still need to -know a referenced type's kind, but can now get it directly off -`ref.index` (via a `List CustomKind.TypeKind` built once, index-aligned -with `project.customTypes`, mirroring gen-sdk's own `List Bool` pattern) -instead of through `Structures/CustomKind.dhall`'s `Model.Name -> TypeKind` -closure and its `Text/equal`-based construction in `buildLookup`. - -Net effect once migrated: `buildLookup`, `Structures/CustomKind.dhall`'s -`Lookup` type, and the `Text/equal` call at -`Interpreters/Project.dhall:601` all go away. `Structures/CustomKind.dhall` -likely still keeps `TypeKind`/`Identity` (renamed/reshaped to be -index-keyed) since `CustomTypeGen.run` and the Python codegen side still -need the className/moduleName/order identity — only the *lookup mechanism* -is replaced, not the whole module. - -Relevant pins for the migration (computed via `dhall hash`, not raw file -`shasum` — Dhall pins are semantic-CBOR hashes, not byte hashes): - -``` --- src/Deps/Contract.dhall -https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall - sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef - --- src/Deps/Sdk.dhall -https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall - sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 -``` - -Both changelogs (`gen-contract` v5.0.0, `gen-sdk` v3.0.0) explicitly call -out the `Scalar.Custom` shape as a **breaking change** — bumping the pin -alone will not compile; every site currently pattern-matching -`Scalar.Custom` with a bare `Name` (`Interpreters/Scalar.dhall:43-51` at -minimum) breaks and needs updating to destructure `CustomTypeRef` instead. -gen-sdk v3.0.0's `Fixtures/Exhaustive.dhall` also grew a domain custom type -and a composite-over-domain type in a non-`public` schema with a -divergent `pgName`, specifically to exercise this — python.gen's own -fixtures/golden tests should be checked against that once the pin bumps. - -Full background, live evidence (reproduced `pgn generate` runs showing the -old Skip-cascade and rank-limit behavior in detail), and the historical -context of why `buildLookup` was withdrawn-then-reinstated live in the -companion note: `/private/tmp/claude-501/-Users-mojojojo-repos-pgenie-python-gen/018e3c75-72c2-414c-af89-0e300db2f020/scratchpad/handoff-buildlookup-text-equal.md` -(session-local scratch path, from the *original* audit session — may -already be cleaned up; copy anything worth keeping into this repo). - -## What the facade un-flattening (item 5) needs to look like - -Not yet designed. The current flat facade -(`tests/golden/src/specimen_client/__init__.py`) does e.g.: - -```python -from ._generated._core import JsonValue as JsonValue -from ._generated.statements.get_specimen import GetSpecimenRow as GetSpecimenRow -``` - -— everything at one flat top level. The replacement needs statements and -types each importable from their own sub-namespace instead, so that e.g. a -custom type named `json_value` and the reserved core symbol `JsonValue` -never land in the same importable namespace, and a query's `FooRow` and an -unrelated custom type also named `foo_row` don't either. Exact shape -(e.g. `from mypackage.statements import get_specimen` / -`from mypackage.types import Mood` vs. some other split) still needs -deciding — this was flagged during the grill as needing design work, not -resolved in detail. - -Note this interacts with the "takeover-ready client" ergonomics work -(`1a5de19`, `ae2869c`) which specifically aimed for a flat, single-import -surface — un-flattening is a deliberate reversal of that goal, traded for -structural collision-safety. `5ac6d49`'s `test_takeover_contract.py` and -`a1ac9f8`'s Ruff-ordering fix both assume the flat shape and will need -rework. - -## What's already done, independent of the above - -- **[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75)** - filed, asking pgn to reject non-normalized query filenames (confirmed - live: `get-specimen.sql` and `get_specimen.sql` both resolve to - `get_specimen`, hyphenated filenames are accepted by pgn on their own — - not rejected upstream) and guarantee unique custom-type identities at the - source. - -## Next steps - -1. ~~Nikita attempts the gen-contract change for item 4.~~ Done — shipped - as `gen-contract` v5.0.0 + `gen-sdk` v3.0.0, both released 2026-07-14. -2. ~~Come back to this document, update item 4's disposition.~~ Done above - — clean reimplementation via gen-sdk's new `CustomTypes` module, not - blocked debt. -3. Bump this branch's pins (`src/Deps/Contract.dhall` → v5.0.0, - `src/Deps/Sdk.dhall` → v3.0.0, hashes above) and fix the resulting - compile breakage at every `Scalar.Custom` pattern match - (`Interpreters/Scalar.dhall` at minimum — grep for other sites once the - bump is in). -4. Migrate `buildLookup`/`effectiveResolvedCustomTypes` - (`Interpreters/Project.dhall:590-611,1174-1204`) onto - `Sdk.CustomTypes.{supportedCustomTypesReasoned, queryIsSupported}`, and - reshape `Structures/CustomKind.dhall`'s `Lookup` from a - `Model.Name -> TypeKind` closure to an index-aligned `List TypeKind` - sourced from `ref.index`. Confirm the rank-limit checks in - `ParamsMember.dhall`/`Member.dhall` still get the kind they need through - the new shape. -5. Only then start implementation on items 5-8: new commits on top of - current `HEAD` (not a history rewrite — the 15 commits' - hypothesis-testing context in their messages is worth keeping). diff --git a/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md b/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md deleted file mode 100644 index b96e2c2..0000000 --- a/docs/plans/2026-07-14-gen-contract-v5-sdk-v3-migration.md +++ /dev/null @@ -1,1061 +0,0 @@ -# Plan: bump gen-contract/gen-sdk pins and retire `buildLookup`'s `Text/equal` - -## Background - -This implements steps 3-4 of the "Next steps" section in -`docs/plans/2026-07-14-detext-equal-branch-filter.md` (item 4 of that -document's decision table). That document is the record of *why*; this -document is the *how*, broken into tasks for subagent-driven execution. -Read the background doc if you want the full history — it is not required -reading for any task below, which are self-contained. - -**Scope grew mid-execution.** Task 1 (bumping gen-contract/gen-sdk pins) -could not be verified against `pgn` v0.9.1 — it rejects gen-contract major -version 5 outright. `pgn` v0.12.0 fixes that, but v0.11.0 (a prerequisite) -dropped the `Text/equal` builtin from its embedded Dhall evaluator entirely. -That forced a live investigation (documented in Task 2's "Why this exists" -and Task 3's "Why this shape") which found that `Text/equal`-free rework is -*impossible* (not just hard) for anything that makes a decision from -comparing two arbitrary runtime strings — which pulled decision-table items -6-7 (same-kind collision detection, rename mappings) and the ADR into scope -now, plus `PythonNamespace.dhall`'s collision detection more broadly (never -part of items 6-7). Task 3 covers all of that. Item 5 (facade -un-flattening) remains **out of scope** — it's unrelated to `Text/equal` and -still needs its own design pass. - -## Global constraints - -- New pins (`dhall hash`-computed, already verified in the background doc): - ``` - -- src/Deps/Contract.dhall - https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall - sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef - - -- src/Deps/Sdk.dhall - https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall - sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 - ``` -- Verification oracle for every task: `mise run test` (drives real `pgn` - subprocesses against the local PostgreSQL server already running on - `localhost:5432`). Also run `mise run lint`. Do **not** invent a separate - `dhall` CLI invocation to typecheck `.dhall` files standalone — the - pgn-fork Dhall evaluator (needed for anything using `Text/equal`, before - Task 3 removes the remaining uses) is only available locally through - `pgn generate` (invoked by the mise tasks). -- Task 1 was written and dispatched before Task 2/3 existed, when `pgn` - v0.9.1 was still pinned — its own text still says "unrelated `Text/equal` - usage" and "out of scope"; that was correct *at the time* for Task 1's - own diff, but Tasks 2-3 below supersede those scoping notes for the - branch as a whole. Follow each task's own instructions as written. -- Do not touch decision-table items 1-3 (`emitSync`, custom-type - registration/`sameOrder`/`order` field, `PyIdent.dhall`'s reserved-name - disambiguation) — they are unrelated and already `Text/equal`-free (aside - from the `py-ident-fix` cleanup pulled in by Task 2, which keeps the same - behavior). -- Regenerate the golden fixture (`mise run golden`) if `pgn generate` output - changes at all (it should not — this migration changes only the Dhall - generator's internals, not its emitted Python), and confirm - `tests/golden` has no diff. If it does diff, treat that as a signal - something changed unintentionally, not something to silently accept. -- Update the "Upcoming" section of `CHANGELOG.md` with one entry describing - the pin bump and the elimination of `buildLookup`'s `Text/equal` (after - Task 4). Follow the existing terse bullet style in that file. - ---- - -## Task 1: Bump pins to gen-contract v5.0.0 / gen-sdk v3.0.0 and fix compile breakage - -### Why this shape - -Bumping the pin is a breaking change independent of the `buildLookup` -migration (Task 4). Two things changed in the contract: - -1. `Scalar.Custom` changed from carrying a bare `Name` to carrying a - `CustomTypeRef`: - ```dhall - let CustomTypeRef = - { name : Name, pgSchema : Text, pgName : Text, index : Natural } - - let Scalar = < Primitive : Primitive | Custom : CustomTypeRef > - ``` - `index` is a 0-based position into `Project.customTypes`, and the - contract now *guarantees* that list is topologically sorted (every index - reachable from `customTypes[i]` is `< i`). `CustomTypeRef` still carries - `.name : Name`, so every site that only used the bare `Name` (e.g. to - feed the *existing* `Text/equal`-based lookup) can keep doing exactly - that by reading `.name` off the ref — this task does **not** change the - lookup mechanism, that is Task 4. - -2. `Value` flattened its optional array wrapper: - ```dhall - -- old (v4.0.1) - let ArraySettings = { dimensionality : Natural, elementIsNullable : Bool } - let Value = { arraySettings : Optional ArraySettings, scalar : Scalar } - - -- new (v5.0.0) - let Value = - { dimensionality : Natural - , elementIsNullable : Bool - , scalar : Scalar - } - ``` - `dimensionality = 0` now means "no array" (replacing - `arraySettings = None ArraySettings`); `elementIsNullable` is meaningless - when `dimensionality = 0`, same as before. There is no more - `Model.ArraySettings` type at all. - -This task's only job is: make every one of these sites compile again, -preserving current behavior exactly. Do not build the `List TypeKind` -lookup replacement here — that is Task 4, and mixing the two makes either -task hard to review independently. - -### Exact changes - -**`src/Deps/Contract.dhall`** — replace both lines with the new pin (URL and -hash both change): -``` -https://raw.githubusercontent.com/pgenie-io/gen-contract/v5.0.0/src/package.dhall - sha256:a1b48fe025c5536b13907bcd2db307cd438f3ae0f67a59222b3aa39e4bdac9ef -``` - -**`src/Deps/Sdk.dhall`** — replace both lines with the new pin: -``` -https://raw.githubusercontent.com/pgenie-io/gen-sdk/v3.0.0/src/package.dhall - sha256:368e4ee1f7557e8a1713a0ac53db6bbfb476027b322d06a660842e5e22e18662 -``` - -**`src/Interpreters/Scalar.dhall`** — `Output.customRef` changes type from -`Optional Model.Name` to `Optional Model.CustomTypeRef`, and the `Custom` -branch now receives a `CustomTypeRef` instead of a bare `Name`: -```dhall -let Output = - { pyType : Text - , imports : ImportSet.Type - , customRef : Optional Model.CustomTypeRef - , decode : ScalarDecode - } - -let run = - \(config : Config) -> - \(input : Input) -> - merge - { Primitive = - \(primitive : Model.Primitive) -> - Lude.Compiled.map - Primitive.Output - Output - ( \(p : Primitive.Output) -> - { pyType = p.pyType - , imports = p.imports - , customRef = None Model.CustomTypeRef - , decode = ScalarDecode.Passthrough - } - ) - (Primitive.run {=} primitive) - , Custom = - \(ref : Model.CustomTypeRef) -> - Lude.Compiled.ok - Output - { pyType = ref.name.inPascalCase - , imports = ImportSet.empty - , customRef = Some ref - , decode = ScalarDecode.Custom - } - } - input -``` -(Only the `customRef` field's type and the `Custom` branch's parameter -changed; `Primitive` branch changes only in the type annotation on -`None Model.CustomTypeRef`.) - -**`src/Interpreters/Value.dhall`** — `input.arraySettings` (an -`Optional Model.ArraySettings`) becomes two plain fields directly on -`Input`. Replace the `Prelude.Optional.fold` over `arraySettings` with a -check on whether `input.dimensionality` is zero: -```dhall -let run = - \(config : Config) -> - \(input : Input) -> - Lude.Compiled.map - Scalar.Output - Output - ( \(scalar : Scalar.Output) -> - if Natural/isZero input.dimensionality - then { pyType = scalar.pyType - , imports = scalar.imports - , scalar - , dims = 0 - , elementIsNullable = False - } - else let elementType = - if input.elementIsNullable - then "${scalar.pyType} | None" - else scalar.pyType - - let arrayType = - Natural/fold - input.dimensionality - Text - (\(inner : Text) -> "list[${inner}]") - elementType - - in { pyType = arrayType - , imports = scalar.imports - , scalar - , dims = input.dimensionality - , elementIsNullable = input.elementIsNullable - } - ) - (Scalar.run {=} input.scalar) -``` -`Output` (python.gen's own internal shape: `pyType`, `imports`, `scalar`, -`dims`, `elementIsNullable`) is unchanged — only how it's derived from -`Input` changes. `qualifyCustom` changes only its type annotation -(`Optional Model.Name` → `Optional Model.CustomTypeRef`) and reads -`.name.inPascalCase` off the ref instead of the bare name directly: -```dhall -let qualifyCustom - : Text -> Text -> Output -> Text - = \(prefix : Text) -> - \(className : Text) -> - \(value : Output) -> - Prelude.Optional.fold - Model.CustomTypeRef - value.scalar.customRef - Text - ( \(ref : Model.CustomTypeRef) -> - Text/replace - ref.name.inPascalCase - (prefix ++ className) - value.pyType - ) - value.pyType -``` - -**`src/Interpreters/Member.dhall`** — the `Custom` branch unwraps -`value.scalar.customRef` (now `Optional Model.CustomTypeRef`). Rename the -bound variable from `name` to `ref` for clarity and feed `ref.name` to -`lookup` (the lookup mechanism itself is untouched in this task — still -`Model.Name -> TypeKind`): -```dhall -, Custom = - Prelude.Optional.fold - Model.CustomTypeRef - value.scalar.customRef - (Lude.Compiled.Type Output) - ( \(ref : Model.CustomTypeRef) -> - let mkOutput = ... -- unchanged body, still uses `identity` - ... - in merge - { Enum = ... -- unchanged - , Composite = ... -- unchanged - , Absent = - Lude.Compiled.report - Output - [ ref.name.inSnakeCase ] - "Custom type not found in project customTypes" - } - (lookup ref.name) - ) - ( Lude.Compiled.report - Output - [ input.pgName ] - "Custom scalar without a customRef name" - ) -``` -Every other reference to the old `name : Model.Name` binding inside this -branch (the two `input.pgName, name.inSnakeCase` error-path lists) becomes -`input.pgName, ref.name.inSnakeCase`. Do not change anything about the -`Enum`/`Composite`/`Absent` merge arms themselves or the dims math. - -**`src/Interpreters/ParamsMember.dhall`** — identical shape of change as -Member.dhall, in the `Prelude.Optional.fold Model.Name value.scalar.customRef -...` block (around line 255-326): rename the bound `name : Model.Name` to -`ref : Model.CustomTypeRef`, change the fold's type argument to -`Model.CustomTypeRef`, call `lookup ref.name`, and update the two -`[input.pgName, name.inSnakeCase]` error-path lists to -`[input.pgName, ref.name.inSnakeCase]`, and the `Absent` arm's -`[name.inSnakeCase]` to `[ref.name.inSnakeCase]`. Also check -`isJsonbScalar`/`isJsonScalar`/`valueIsArray`/`scalarIsJson` earlier in this -file (used to compute `needsJsonbImport`/`needsJsonImport`/ -`isJsonArrayParam`) for any direct reads of `value.arraySettings` or -`Model.ArraySettings` — if present, adapt them the same way as Value.dhall -(`dimensionality`/`elementIsNullable` fields directly on `Model.Value` -instead of an `Optional Model.ArraySettings` wrapper). Read the full file -before editing; the excerpt in this brief only covered lines 200-339. - -**`src/Interpreters/CustomType.dhall`** — two things: -1. The `Composite` branch's member loop matches on `m.value.scalar` with - `Custom = \(name : Model.Name) -> ...` (checking - `m.value.arraySettings` to reject "Custom array fields inside a - composite type are not supported"). Update the merge arm's bound - variable type to `Model.CustomTypeRef` (rename `name` to `ref`, - references become `ref.name`/`ref.name.inSnakeCase`), and change the - `Optional Model.ArraySettings`/`m.value.arraySettings` check to - `Prelude.Bool.not (Natural/isZero m.value.dimensionality)` (i.e. "has at - least one array dimension") in place of testing whether the `Optional` - is `Some`. -2. `lookup input.name` (the self-consistency check, `input : Model.CustomType`) - is unaffected — `Model.CustomType.name` is still a plain `Model.Name`, - untouched by this contract version's breaking changes. Leave it as-is - in this task. - -**`src/Interpreters/Result.dhall`, `src/Interpreters/ResultColumns.dhall`, -`src/Interpreters/Query.dhall`** — these only carry `lookup : CustomKind.Lookup` -through as an opaque parameter (grep confirms no direct `Model.Name`/ -`Model.ArraySettings` access in these three files). They should not need any -edits; if `dhall`/`pgn generate` reports an error originating in one of -these files, that means this brief's premise was wrong somewhere upstream — -investigate rather than patching around it here. - -**`src/Interpreters/Project.dhall`** — do **not** touch `buildLookup`, -`effectiveResolvedCustomTypes`, or `CustomKind.Lookup` in this task (that's -Task 4). The only things in this file that might need attention from the -pin bump: grep the file yourself for any other `Model.Name`/ -`Model.ArraySettings` reads this brief didn't enumerate (the `Custom` merge -arm inside `resolveCustomTypes`'s `kind` computation, for instance, matches -on `customType.definition`, not `Scalar.Custom`, so it should be unaffected -— confirm this rather than assuming it). - -### Also check - -Run `grep -rn "Model.ArraySettings\|arraySettings" src/` yourself after -reading each file above — this brief's line numbers may drift slightly from -exact current file content. Fix every site the grep turns up, not just the -ones enumerated here. - -### Verification - -1. `mise run test` — must pass (73 passed, 0 skipped per the README's - documented baseline; note the exact count down if it differs and - investigate why before treating that as a pass). -2. `mise run lint`. -3. `mise run golden`, then `git diff tests/golden` — expect **no diff**. - If there is a diff, do not commit it silently; report it as a concern. -4. Report the exact commands run and their output in the task report. - -### Report file - -Write your report to the path given in the dispatch prompt. Include: which -files you touched beyond this brief's list (if any, and why), the full -`mise run test` summary line, and confirmation of a clean `git diff -tests/golden`. - ---- - -## Task 2: Bump pgn to v0.12.0, pull in the `py-ident-fix` cleanup, re-verify Task 1 - -### Why this exists - -Task 1 landed correctly (commit `d18398c`) but could not be verified: -`pgn` v0.9.1 (pinned in `mise.toml`) hard-rejects gen-contract major version 5 -("Incompatible contract major version: 5. Expected 4."). `pgn` v0.12.0 -(released 2026-07-14) adds gen-contract v5.0.0 support — confirmed via its -changelog: *"Bump the `gen-contract` dependency to v5.0.0... Add -`CustomTypeRef`... Flatten `Value`'s `arraySettings`..."*, exactly matching -Task 1's changes. - -`pgn` v0.11.0 (a prerequisite of v0.12.0) is a breaking change: -*"Update to Dhall that lacks `Text/equal` to stay in line with the official -Dhall spec."* Confirmed empirically (both against the real `pgn` fork via a -throwaway `pgn generate` run, and against stock upstream `dhall` 1.42.3): -`Text/equal`, `Text/length`, and `Bool/equal` are all unavailable — there is -no way to produce a `Bool` or a differently-typed decision (e.g. an `Ok`/`Err` -union) from comparing two arbitrary runtime `Text` values using only -`Text/replace`. `Text/replace`-based tricks (this repo's old digit-marker -trick, and gen-sdk's own `Lude.Text.replaceIfEqual`/`replaceIfOneOf`) only -ever produce more `Text` — they can transform text, not decide anything. This -is why Task 3 (below) has to discard rather than "rework" several checks. - -Branch `py-ident-fix` (already pushed, commit `7685c0a8794b0d4e20947c2ff6aa9c11b4dabd48`, -"python.gen: replace PyIdent's hand-rolled keyword equality with -Lude.Text.replaceIfOneOf") already did the one part of this that *is* just a -transform (not a decision): `PyIdent.dhall`'s Python-keyword-suffixing. Pull -that commit's change into this branch as part of this task — it is a strict -simplification of code Task 1 already touches, with the same public -interface and unchanged golden output per its own commit message. - -### Exact changes - -1. **Cherry-pick or manually reapply** commit `7685c0a8794b0d4e20947c2ff6aa9c11b4dabd48` - from `py-ident-fix` onto this branch (`git log py-ident-fix` to find it; - `git cherry-pick 7685c0a8794b0d4e20947c2ff6aa9c11b4dabd48` is the - straightforward route — resolve any conflict against Task 1's changes by - keeping Task 1's contract-type changes and `py-ident-fix`'s - `sanitizeAgainst`/`Lude.Text.replaceIfOneOf` rewrite; they don't overlap - in `PyIdent.dhall`, which Task 1 never touched). -2. **`mise.toml`**: change the pgn tool pin from `v0.9.1` to `v0.12.0`: - ```toml - "github:pgenie-io/pgenie" = { version = "v0.12.0", exe = "pgn" } - ``` - Run `mise install` (or just let the next `mise run` task trigger it) to - fetch the new binary. -3. Re-run Task 1's verification now that the environment is unblocked: - `mise run test`, `mise run lint`, `mise run golden` + `git diff - tests/golden` (expect no diff). Do not make further Dhall changes in this - task beyond the cherry-pick above — if `mise run test` still fails for a - reason unrelated to the pgn version (e.g. a genuine bug in Task 1's - compile fixes), report it as a concern rather than silently patching - Task 1's work under this task's commit. - -### Verification - -`mise run test`, `mise run lint`, `mise run golden` + `git diff -tests/golden` (expect no diff). Report the exact `mise run test` summary -(pass/fail counts) — this is the first time it can actually run end to end -since Task 1 started, so capture it precisely. - -### Report file - -`.superpowers/sdd/task-2-report.md`. State clearly: the cherry-picked commit -hash (or, if reapplied manually, why), the new `mise run test` result, and -confirmation of clean `git diff tests/golden`. - ---- - -## Task 3: Discard `Text/equal`-dependent validation (mappings, identity check, namespace collision detection) - -**Depends on Task 2** (needs `pgn` v0.12.0 actually working, so this task's -own verification is meaningful). Do not start until Task 2's review is clean. - -### Why this shape - -With `Text/equal` gone from `pgn`'s embedded Dhall evaluator (confirmed in -Task 2's background), every piece of this generator that makes a *decision* -by comparing two arbitrary runtime `Text` values is no longer expressible — -not "hard to rework," genuinely impossible in the target language (see -Task 2's background section for the proof). Three clusters of code do this: - -1. **`queryNameMappings`/`customTypeNameMappings`** (the rename-mapping - config feature): `PythonNameMapping.dhall`'s `resolveQuery`/ - `resolveCustomType` match a mapping's `source` against a query/type name - via `Text/equal`. This is decision-table item 7 from the background doc - (`docs/plans/2026-07-14-detext-equal-branch-filter.md`) — already marked - **discard entirely**, independent of this new finding. -2. **`validateCustomTypeIdentities`** (`Interpreters/Project.dhall`, checks - whether two different custom types collapse to the same unqualified - contract `Name`): matches decision-table item 6's philosophy exactly — - "pushed upstream instead," and - [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) - (already filed) explicitly asks pgn to "guarantee unique custom-type - identities at the source." Discard, same rationale as item 6. -3. **`PythonNamespace.dhall`'s `validate`** (4 call sites in - `Interpreters/Project.dhall`: per-query local param/result-field - namespace, per-custom-type local field namespace, the project-wide - facade/module namespace, and per-type module-internal bindings): this - goes beyond items 6-7 (the local audits were never proposed for - discard), but it's equally impossible now. Confirmed there is a working - safety net one layer down: `tests/test_generated.py`'s - `test_generated_passes_basedpyright_strict` (and the same pattern in - `test_identifier_collisions.py`, `test_psycopg_adapter_contract.py`, - `test_unsupported_types.py`) already asserts - `errorCount == 0 and warningCount == 0` on the generated package. Verified - empirically: a duplicate dataclass field name triggers basedpyright's - `reportRedeclaration` (warning — fails the gate), and a field name that - shadows a type needed elsewhere triggers `reportInvalidTypeForm` (hard - error). The tradeoff is real and accepted deliberately: today a collision - fails fast at `pgn generate` with a message pointing at the SQL/schema - source; after this task, the same collision still fails the test suite, - just later (at basedpyright time) and less precisely attributed - (pointing at generated Python, not SQL). This was an explicit, discussed - tradeoff, not an oversight — do not try to preserve the old error - messages by some other means. - -### Exact changes - -**Delete entirely:** -- `src/Structures/PythonNameMapping.dhall` -- `src/Structures/PythonNamespace.dhall` -- `docs/adr/0001-generated-python-name-collisions.md` (its entire content - describes the mechanism being removed in this task — stale documentation - actively describing a removed feature is worse than no doc at all; do not - wait for item 8's broader doc pass) -- `tests/test_python_name_collisions.py` (this file, per the background - doc, "specifically tests the collision-detection and rename-mapping - mechanisms being discarded") - -**`src/package.dhall`**: remove `queryNameMappings`/`customTypeNameMappings` -from `Config` and `Config/default`, and the `PythonNameMapping` import. - -**`src/Interpreters/Project.dhall`**: remove -- `queryNameMappings`/`customTypeNameMappings` from `Config` and - `ResolvedConfig`, and their resolution in `run`. -- `validateCustomTypeIdentities`, `validateQueryMappings`, - `validateCustomTypeMappings`, `finishMappingValidation` (if nothing else - uses it after the other three are gone — check), `QueryMappingState`, - `CustomTypeMappingState`, `CustomTypeIdentity`, `CustomTypeIdentityState`, - and the `mappingsValid`/`mappingsAndLocalsValid` wiring in `run` that - chains them together (fold their removal into whatever the remaining - validation chain becomes — read the full `run` function first to see - what `mappingsAndLocalsValid`/`combined` looks like once - `validateLocalNamespaces` is *also* gone, see next point, since both - disappear in the same task). -- `validateLocalNamespaces`, `validateProjectNamespaces`, and the - `moduleValidation` block inside it — all of `PythonNamespace.validate`'s - 4 call sites. Remove the `PythonNamespace` import. -- The `PythonNameMapping` import, and any leftover reference to the - mapping/namespace types in this file's other functions (grep after - editing to confirm nothing dangles). - -**`src/Interpreters/Query.dhall`** and **`src/Interpreters/CustomType.dhall`**: -remove `queryNameMappings`/`customTypeNameMappings` from their `Config` -types, remove the `PythonNameMapping` import, and replace the -`PythonNameMapping.resolveQuery`/`resolveCustomType` call with the bare -default name directly — e.g. in `CustomType.dhall`: -```dhall --- before -let pythonName = - PythonNameMapping.resolveCustomType - config.customTypeNameMappings - { schema = input.pgSchema, name = input.pgName } - { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase - , pascalCase = PyIdent.pySafeName input.name.inPascalCase - } - --- after -let pythonName = - { snakeCase = PyIdent.typeModuleSafeName input.name.inSnakeCase - , pascalCase = PyIdent.pySafeName input.name.inPascalCase - } -``` -and the analogous simplification in `Query.dhall` around its -`PythonNameMapping.resolveQuery` call. Read both call sites in full before -editing — this brief doesn't reproduce their exact surrounding code. - -**`CustomType.dhall`'s `moduleBindings`/namespace-binding plumbing** -(`enumNamespaceBindings`, `compositeNamespaceBindings`, `moduleNamespace`, -`moduleBinding`, and the `Output.moduleBindings` field): these exist solely -to feed `PythonNamespace.validate`'s per-module check (now gone). Remove -them and the `moduleBindings` field from `CustomType.dhall`'s `Output`, and -its consumption in `Project.dhall`. Confirm nothing else reads -`moduleBindings` before deleting it (grep first). - -**Also check** `Interpreters/ResultColumns.dhall`, `Result.dhall`, -`Member.dhall`, `ParamsMember.dhall` for any `PythonNamespace`/ -`PythonNameMapping` references this brief didn't enumerate (expected: none, -but confirm). - -### Tests - -- Delete `tests/test_python_name_collisions.py` (see above). -- Read `tests/test_identifier_collisions.py` in full before touching it: it - covers query-name-vs-implementation-global collisions (decision-table - item 3's cluster, `PyIdent.dhall`'s reserved-name disambiguation — kept, - not part of this task) and *also* uses basedpyright directly (lines - ~282-290). Update only the parts of it that exercise the - now-deleted `queryNameMappings`/`customTypeNameMappings`/ - `PythonNamespace`-based paths, if any — keep everything about reserved - implementation-name disambiguation as-is. -- Read `tests/test_unsupported_types.py` in full — it references - `PythonNameMapping`/mapping config (per the earlier grep). Update it to - drop mapping-specific scenarios while keeping its `onUnsupported: Skip` - coverage intact (that coverage matters for Task 4, next). -- **`tests/test_unsupported_types.py`'s `_write_contract_probe` helper** - (around line 115, feeding the 6-case parametrized - `test_custom_shape_contracts_fail_loudly`) hand-authors a synthetic Dhall - wrapper module as a Python string, and needs updating for *this* task's - changes specifically (Task 4 will need a second pass on the same helper - for its own changes — see Task 4's Tests section below, don't try to - anticipate that part here): - - `interpreter_config` currently emits - `{ customTypeNameMappings = [] : List PythonNameMapping.CustomType }` - when `interpreter == "CustomType"` — `PythonNameMapping` no longer - exists after this task. Since `CustomType.dhall`'s `Config` also drops - `customTypeNameMappings` in this task (per the `Interpreters/*.dhall` - changes above), the interpreter config becomes `{=}` unconditionally - (same as the non-`CustomType` branch) — the `if interpreter == - "CustomType" else` branch can go entirely. - - The wrapper's own `let PythonNameMapping = ./Structures/PythonNameMapping.dhall` - import line must be removed. - - This probe is still on the *old* contract shape from before Task 1 (it - was never updated when Task 1 landed, since Task 1 didn't know test - fixtures existed at this level — confirmed by code review): its - `member.value` still builds `{ arraySettings = {array_settings}, scalar - = Model.Scalar.Custom name }` where `array_settings` renders - `"None Model.ArraySettings"` or `"Some { dimensionality = N, - elementIsNullable = False }"`, and `Model.Scalar.Custom name` passes a - bare `Model.Name`. Both no longer typecheck against the now-bumped - gen-contract v5.0.0 pin (Task 1). Fix this probe to the new shape as - part of *this* task (it's blocking verification regardless of which - task's changes it's nominally testing): `value` becomes - `{ dimensionality = {dimensionality}, elementIsNullable = False, scalar - = Model.Scalar.Custom { name, pgSchema = "public", pgName = - "probe_value", index = 0 } }` (drop the `array_settings` local - entirely, inline `dimensionality` directly — `0` for the non-array - case, same as today's `None` case). Confirm the probe still produces - the same 6 pass/fail outcomes the test asserts. - - Leave the probe's `CustomKind.Lookup` construction - (`\(_ : Model.Name) -> {lookup}`) as-is in this task — that's Task 4's - job, since `CustomKind.Lookup`'s shape doesn't change until then. -- Do **not** touch `tests/test_takeover_contract.py` — it's about the flat - facade (item 5), out of scope for this whole plan. -- Run the full suite; if new failures appear that trace to bindings this - task didn't anticipate removing, investigate rather than papering over - them (e.g. with a stub mapping). - -### Docs - -- **`README.md`**: remove the `queryNameMappings`/`customTypeNameMappings` - rows from the config table (lines ~46-47 as of this writing); remove the - "Generated Python namespaces are audited..." paragraph and its YAML - mapping example (~lines 62-96, read current content — line numbers will - have drifted from Task 1/2's edits); remove the ADR 0001 link. Replace - with a short, honest paragraph: namespace collisions are not detected at - generation time; a colliding schema fails `basedpyright --strict` on the - generated package instead (link to - [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) for - the upstream ask to prevent custom-type identity collisions at the - source). Don't overclaim — collisions are *possible*, just caught later. -- **`DESIGN.md`** section 5 ("Configuration, mapping, and unsupported - shapes"): remove the mapping-specific config fields/description and the - ADR reference; keep the `onUnsupported` description (unrelated). Section - 10 ("The pinned `Text/equal` constraint"): this section is also rewritten - by Task 4 for `buildLookup` specifically — in *this* task, update it (or - leave a marker for Task 4) to note that mapping/namespace/identity - validation no longer uses `Text/equal` either, and enumerate what (if - anything) still does after this task (check with a repo-wide grep for - `Text/equal` once done — there should be very little, if anything, left - outside `Interpreters/Project.dhall`'s `buildLookup`, which Task 4 - removes next). -- **`CHANGELOG.md`**: add an "Upcoming" bullet (matching the file's terse - style) noting the removal of `queryNameMappings`/`customTypeNameMappings` - and generation-time namespace-collision detection, with the basedpyright - rationale, and a bullet for the pgn v0.12.0 bump (Task 2, if not already - added there). Also fix the now-stale released-history bullet "Kept - `PyIdent.dhall` independent of fork-only text equality by using its - bounded `Text/replace` marker construction" (~line 82 as of this - writing) — Task 2's cherry-pick already replaced that exact mechanism - with `Lude.Text.replaceIfOneOf`. Don't rewrite released history in place; - either amend that bullet's wording to describe the current mechanism - accurately (it's still describing *current* behavior, just the wrong - implementation detail) or add a one-line "Upcoming" bullet superseding it - — whichever fits this file's existing convention for describing a - changed implementation detail of already-released behavior. - -### Verification - -`mise run test`, `mise run lint`, `mise run golden` + `git diff -tests/golden` (a diff here IS expected in this task, since removing -mapping config could change nothing about default-path output, but confirm -this — if `tests/golden` changes, explain exactly why in the report). - -### Report file - -`.superpowers/sdd/task-3-report.md`. Include: full file list touched, the -`mise run test` summary, confirmation of `tests/golden` status (diff or no -diff, with explanation either way), and a repo-wide `grep -rn "Text/equal" -src/` output so the next task (Task 4) knows exactly what's left. - ---- - -## Task 4: Replace `buildLookup`/`effectiveResolvedCustomTypes` with `Sdk.CustomTypes` - -**Depends on Tasks 1-3** (needs the bumped pins, working `pgn` v0.12.0, and -`Model.CustomTypeRef` plumbing already in place). Do not start this task -until Task 3's review is clean. - -### Why this shape - -`gen-sdk` v3.0.0 ships a ready-made, `Text/equal`-free replacement for -python.gen's own removal-cascade, in `Sdk.CustomTypes`: - -```dhall --- List Bool: index i tells whether customTypes[i] is supported, given a --- caller-supplied "is this kind of definition supported at all" predicate. --- Composite/Domain members' own Custom refs are checked transitively via --- List/fold over customTypes in order, since customTypes is topologically --- sorted (every ref.index < the referencing type's own index). -supportedCustomTypes - : (Contract.CustomTypeDefinition -> Bool) -> - List Contract.CustomType -> List Bool - --- Same fold, but returns Some instead of False --- for each unsupported index (useful for warning messages). -supportedCustomTypesReasoned - : (Contract.CustomTypeDefinition -> Bool) -> - List Contract.CustomType -> List (Optional Contract.CustomTypeRef) - -customTypeIsSupported : List Bool -> Contract.CustomTypeRef -> Bool -queryIsSupported : List Bool -> Contract.Query -> Bool -``` - -This replaces the *removal-cascade* half of `buildLookup` + -`effectiveResolvedCustomTypes` (the repeated `Natural/fold` over -`Text/equal`-keyed lookups). It does **not** replace python.gen's own -rank-limit checks (array-of-enum >2 dims, array-of-composite >1 dim) — -those are python.gen-specific and stay, but they can now be expressed using -a referenced type's *kind* (Enum/Composite/Domain) obtained directly via -`ref.index`, instead of `Structures/CustomKind.dhall`'s -`Model.Name -> TypeKind` closure. - -**The subtlety that makes this not a mechanical swap:** the *kind* -classification of a custom type (is it an Enum, a Composite, or a Domain) -never changes across removal passes — it's a fixed fact about each type's -own definition. The *old* `buildLookup` conflated two separate concerns by -shrinking its entry list each pass: (a) "what kind is this referenced type" -and (b) "is this referenced type still a survivor" — a lookup miss meant -`Absent` regardless of which of the two failed, and `Absent` is what causes -a downstream compile to reject the reference (`"Custom type not found in -project customTypes"`). The new design must keep producing that same -`Absent`-on-either-failure behavior, or a composite that depends on a -removed type will wrongly keep compiling (it would still find the removed -type's *real* kind via a fixed classification, silently referencing a type -that no longer gets emitted). Concretely: build the kind classification -once (fixed, never shrinks), build the supported/removed cascade once (via -`Sdk.CustomTypes`), then mask the two together — a reference to an index -that is not "supported" reads as `Absent` regardless of its real kind, only -in `Skip` mode (in `Fail` mode nothing is masked, matching current -behavior exactly: `Fail` never drops anything, so every input is looked up -at its real kind and only fails via one `CustomTypeGen.run`/`Member`/ -`ParamsMember` error path, same as today). - -### Exact changes - -**`src/Structures/CustomKind.dhall`** — reshape `Lookup` from a closure to -an index-aligned list, and add an `at` accessor (mirrors gen-sdk's own -`optionalIndex` helper in `supportedCustomTypes.dhall`): -```dhall -let Model = ../Deps/Contract.dhall - -let Prelude = ../Deps/Prelude.dhall - -let Identity = - { className : Text, moduleName : Text, order : Natural } - -let TypeKind = - < Enum : Identity - | Composite : Identity - | Absent - > - -let Lookup = List TypeKind - -let at = - \(lookup : Lookup) -> - \(index : Natural) -> - Prelude.Optional.fold - TypeKind - (Prelude.List.index index TypeKind lookup) - TypeKind - (\(kind : TypeKind) -> kind) - TypeKind.Absent - -in { TypeKind, Lookup, Identity, at } -``` - -**Every call site that currently does `lookup ` where the lookup -was keyed by a `Model.Name`** switches to `CustomKind.at lookup ` -where `` comes from a `Model.CustomTypeRef.index` already in scope -(Task 1 already renamed the relevant bound variables to `ref`): - -- `src/Interpreters/Member.dhall`: `(lookup ref.name)` → `(CustomKind.at lookup ref.index)`. -- `src/Interpreters/ParamsMember.dhall`: `(lookup ref.name)` → `(CustomKind.at lookup ref.index)`. -- `src/Interpreters/CustomType.dhall`: - - The composite-member loop's `Custom = \(ref : Model.CustomTypeRef) -> ...` branch: - `(lookup ref.name)` → `(CustomKind.at lookup ref.index)`. - - The **self**-lookup (`lookup input.name`, appearing twice — once in the - `Enum` merge arm, once in the `Composite` merge arm of `run`) needs the - custom type's *own* index, which `Model.CustomType` does not carry - directly (unlike a *reference* to one). Add an explicit `index : Natural` - parameter to `CustomTypeGen.run`, threaded in by its caller - (`Interpreters/Project.dhall`, see below), and use - `CustomKind.at lookup index` in place of `lookup input.name` in both - merge arms: - ```dhall - let run = - \(config : Config) -> - \(lookup : CustomKind.Lookup) -> - \(index : Natural) -> - \(input : Input) -> - ... - in merge - { Enum = \(identity : CustomKind.Identity) -> ... - , Composite = \(_ : CustomKind.Identity) -> ... - , Absent = ... - } - (CustomKind.at lookup index) - ``` - (Both the `Enum`-definition branch's merge and the `Composite`-definition - branch's merge get this same treatment — each currently ends with - `(lookup input.name)`.) - -**`src/Interpreters/Project.dhall`** — this is where the real restructuring -happens. Replace `buildLookup`, `lookupEntries`'s dependents, and -`effectiveResolvedCustomTypes` as follows. Keep `IndexedCustomType`, -`ResolvedCustomType`, `LookupEntry`, `LookupKind`, `resolveCustomTypes`, and -`lookupEntries` exactly as they are today (they still correctly compute, -per-type, its Python identity and Enum/Composite/Domain kind tag, in -`input.customTypes` order — `resolveCustomTypes` already builds this via -`Prelude.List.indexed Model.CustomType customTypes`, so `entry.index` is -already the same 0-based position space as the contract's `ref.index`, -confirmed same order since `resolveCustomTypes` is called directly with -`input.customTypes`, never a reordered copy). - -Remove `buildLookup` entirely (lines ~590-611) and replace it with: - -```dhall -let kindOf - : List ResolvedCustomType -> CustomKind.Lookup - = \(entries : List ResolvedCustomType) -> - Prelude.List.map - ResolvedCustomType - CustomKind.TypeKind - ( \(rt : ResolvedCustomType) -> - merge - { Composite = CustomKind.TypeKind.Composite rt.lookupEntry.identity - , Enum = CustomKind.TypeKind.Enum rt.lookupEntry.identity - , Domain = CustomKind.TypeKind.Absent - } - rt.lookupEntry.kind - ) - entries - -let dimsAtMostTwo = - \(dims : Natural) -> Natural/isZero (Natural/subtract 2 dims) - -let dimsAtMostOne = - \(dims : Natural) -> Natural/isZero (Natural/subtract 1 dims) - --- Own-definition support check fed to Sdk.CustomTypes.supportedCustomTypesReasoned. --- `fixedKindOf` is the *unmasked* kind classification (kindOf above) — always --- present regardless of survivorship, since kind never changes across passes; --- only the array-rank check depends on it here. -let rankChecked - : CustomKind.Lookup -> Model.CustomTypeDefinition -> Bool - = \(fixedKindOf : CustomKind.Lookup) -> - \(definition : Model.CustomTypeDefinition) -> - merge - { Composite = - \(members : List Model.Member) -> - Prelude.List.all - Model.Member - ( \(member : Model.Member) -> - merge - { Primitive = \(_ : Model.Primitive) -> True - , Custom = - \(ref : Model.CustomTypeRef) -> - merge - { Enum = - \(_ : CustomKind.Identity) -> - dimsAtMostTwo member.value.dimensionality - , Composite = - \(_ : CustomKind.Identity) -> - dimsAtMostOne member.value.dimensionality - , Absent = True - } - (CustomKind.at fixedKindOf ref.index) - } - member.value.scalar - ) - members - , Enum = \(_ : List Model.EnumVariant) -> True - , Domain = \(_ : Model.Value) -> False - } - definition - -let boolAt = - \(bools : List Bool) -> - \(index : Natural) -> - Prelude.Optional.fold - Bool - (Prelude.List.index index Bool bools) - Bool - (\(b : Bool) -> b) - False -``` - -Then in `run`, replace the `effectiveResolvedCustomTypes`/ -`effectiveCustomTypes`/`lookup` block (currently lines ~1166-1204) with: - -```dhall - let resolvedCustomTypes = - resolveCustomTypes - resolvedConfig.customTypeNameMappings - input.customTypes - - let fixedKindOf = kindOf resolvedCustomTypes - - let supported - : List Bool - = if skip - then Prelude.List.map - (Optional Model.CustomTypeRef) - Bool - (Prelude.Optional.null Model.CustomTypeRef) - ( Sdk.CustomTypes.supportedCustomTypesReasoned - (rankChecked fixedKindOf) - input.customTypes - ) - else Prelude.List.map - ResolvedCustomType - Bool - (\(_ : ResolvedCustomType) -> True) - resolvedCustomTypes - - let lookup - : CustomKind.Lookup - = if skip - then Prelude.List.map - { index : Natural, value : CustomKind.TypeKind } - CustomKind.TypeKind - ( \(e : { index : Natural, value : CustomKind.TypeKind }) -> - if boolAt supported e.index then e.value else CustomKind.TypeKind.Absent - ) - (Prelude.List.indexed CustomKind.TypeKind fixedKindOf) - else fixedKindOf - - let indexedResolvedCustomTypes - : List { index : Natural, value : ResolvedCustomType } - = Prelude.List.indexed ResolvedCustomType resolvedCustomTypes - - let effectiveResolvedCustomTypes - : List { index : Natural, value : ResolvedCustomType } - = if skip - then Prelude.List.filter - { index : Natural, value : ResolvedCustomType } - (\(e : { index : Natural, value : ResolvedCustomType }) -> boolAt supported e.index) - indexedResolvedCustomTypes - else indexedResolvedCustomTypes - - let effectiveCustomTypes - : List { index : Natural, value : Model.CustomType } - = Prelude.List.map - { index : Natural, value : ResolvedCustomType } - { index : Natural, value : Model.CustomType } - ( \(e : { index : Natural, value : ResolvedCustomType }) -> - { index = e.index, value = e.value.value } - ) - effectiveResolvedCustomTypes -``` - -This changes `effectiveCustomTypes`'s element type from -`List Model.CustomType` to `List { index : Natural, value : Model.CustomType }` -everywhere it's consumed further down in `run`, so update: - -- `typesForCombine` — was - `Lude.Compiled.traverseList Model.CustomType CustomTypeGen.Output (\(ct : Model.CustomType) -> CustomTypeGen.run customTypeConfig lookup ct) effectiveCustomTypes` - becomes - ```dhall - Lude.Compiled.traverseList - { index : Natural, value : Model.CustomType } - CustomTypeGen.Output - ( \(e : { index : Natural, value : Model.CustomType }) -> - CustomTypeGen.run customTypeConfig lookup e.index e.value - ) - effectiveCustomTypes - ``` - (note the extra `e.index` argument, matching `CustomTypeGen.run`'s new - signature from this task's `CustomType.dhall` change above). -- Anywhere else `effectiveCustomTypes` is used as a plain - `List Model.CustomType` (check `combineOutputs`'s call and any other - reference in `run` — read the current file to find every use before - editing) needs either the same `{index,value}` shape or a - `Prelude.List.map ... (\(e) -> e.value) effectiveCustomTypes` to recover - the bare list, whichever the specific call site needs. `combineOutputs` - itself takes `customTypes : List CustomTypeGen.Output` (already-compiled - output, not `Model.CustomType`), which is unaffected — only the - *pre-compile* `effectiveCustomTypes`/`typesForCombine` wiring changes. - -**Remove entirely** (no longer needed, replaced by the above): -`typeSucceedsWith`, `typeWarningWith`'s use of a per-pass -`candidateLookup` parameter, and the `Natural/fold`-based -`effectiveResolvedCustomTypes` loop. If `skipWarnings` (further down in -`run`) still needs per-type warning `Report`s for `Skip` mode, rebuild it -using the *final* `lookup` (not a shrinking candidate) — i.e. -`typeWarningWith` becomes a plain function of `lookup` (no longer needing a -`candidateLookup` argument at all, since there is only one `lookup` now): -```dhall -let typeWarning = - \(ct : Model.CustomType) -> - \(index : Natural) -> - merge - { Ok = - \(_ : { value : CustomTypeGen.Output, warnings : List Report }) -> - None Report - , Err = - \(err : Report) -> Some { path = [ ct.name.inSnakeCase ] # err.path, message = err.message } - } - (CustomTypeGen.run customTypeConfig lookup index ct) -``` -and its use in `skipWarnings` maps over the **indexed** `input.customTypes` -(so each type is checked at its own real index, not filtered) — mirror -whatever indexing helper you already introduced above rather than -duplicating `Prelude.List.indexed` a third time. - -### Docs to update in this task - -- `README.md`: the paragraph starting "`fixtures/Exhaustive.dhall` is the - contract fixture. It and `buildLookup` rely on the pgn fork's `Text/equal` - builtin..." — `buildLookup` no longer exists or uses `Text/equal`. Correct - this paragraph to describe the current state: `fixtures/Exhaustive.dhall` - itself doesn't use `Text/equal`, but the generator's mapping-validation - code (`queryNameMappings`/`customTypeNameMappings`, - `validateCustomTypeIdentities`, `PyIdent.dhall`) still does, which is why - CI still needs the fork-aware evaluator. Don't claim `Text/equal` is gone - from the repo — it isn't (that's items 6-7, out of scope here). -- `DESIGN.md` section 10, "The pinned `Text/equal` constraint" — this - section specifically describes `buildLookup`'s mechanism, which this task - deletes. Rewrite it to describe the new `Sdk.CustomTypes`-based cascade - (or fold its content into a short note that `buildLookup` was replaced by - `gen-sdk`'s `CustomTypes` module and point at the remaining `Text/equal` - users listed above) — do not just delete the section silently; the - surrounding sections may reference it. -- `DESIGN.md` around line 278/289 (the `-> buildLookup(custom types)` pseudo - pipeline sketch, and "`onUnsupported: Skip` repeatedly rebuilds - `buildLookup`...") — update to describe the new one-pass - `Sdk.CustomTypes.supportedCustomTypesReasoned` mechanism instead of a - repeated rebuild. -- `CHANGELOG.md`: the existing "Retained `buildLookup`..." bullet (in a - past/released section, not "Upcoming") describes a past decision that no - longer holds. Do not edit released history; instead add a new bullet under - "Upcoming" (per Global Constraints) noting the v5.0.0/v3.0.0 bump and that - `buildLookup` was replaced by `gen-sdk`'s `CustomTypes` module. -- Leave `docs/upstream-asks.md` alone unless you find it makes a factual - claim about `buildLookup`'s *current* mechanism (as opposed to the - upstream ask itself, which is unrelated to this task) — read it first to - check. - -### Tests - -`tests/test_unsupported_types.py`'s `_write_contract_probe` helper (around -line 115, feeding the parametrized `test_custom_shape_contracts_fail_loudly`) -hand-authors a synthetic Dhall wrapper module as a Python string. Task 3 -already updated it for the v5 contract shape and removed its -`PythonNameMapping` reference; this task needs one more pass, since -`CustomKind.Lookup` changes shape here: -- Its `let lookup : CustomKind.Lookup = \(_ : Model.Name) -> {lookup}` - construction (a `Model.Name -> TypeKind` closure) must become a - `List CustomKind.TypeKind` — for these probes there's exactly one custom - type in play ("ProbeValue"), so `lookup` becomes a one-element list: - `let lookup : CustomKind.Lookup = [ {lookup} ]`. -- `CustomType.dhall`'s `run` gained an explicit `index : Natural` parameter - in this task (the self-lookup consistency check). The probe's call site - (`Target.run interpreterConfig lookup {target_input}`) needs an index - argument threaded in when `interpreter == "CustomType"` — `0` (matching - the single-element `lookup` list above). For the `Member`-interpreter - probes (`interpreter == "Member"`), `Member.dhall`'s `run` signature is - unaffected by this task (it never needed a self-index, only the - already-present `ref.index` off the value being looked up) — don't add - an index argument there. -- Confirm all cases of `test_custom_shape_contracts_fail_loudly` still - produce the same pass/fail outcomes and error messages after this - change — this is the test most likely to catch a subtle regression in - the new `Absent`-on-either-failure masking logic (see this task's "Why - this shape" section above). - -### Verification - -Same as Task 1: `mise run test` (expect the same passing count as Task 1 -left it at), `mise run lint`, `mise run golden` + `git diff tests/golden` -expecting no diff. Additionally: since this task touches the `Skip`-mode -cascade specifically, if this repo's test suite exercises -`onUnsupported: Skip` (check `tests/` for it), pay particular attention to -those tests passing — this is the behavior most at risk of a subtle -regression from this refactor (see the "Absent-on-either-failure" subtlety -above). - -### Report file - -Same contract as Task 1: report which files changed, the full `mise run -test` summary, confirmation of clean `git diff tests/golden`, and -explicitly confirm whether any `Skip`-mode-specific test exists and passed. From 2eed9cc51aab335560ba9f110500e66ece135123 Mon Sep 17 00:00:00 2001 From: Viacheslav Shvets Date: Thu, 16 Jul 2026 13:24:16 +0300 Subject: [PATCH 14/14] Fix PR migration checks and collision docs --- .github/workflows/ci.yml | 18 ++++------ CHANGELOG.md | 28 ++++++--------- DESIGN.md | 23 +++++++------ README.md | 17 ++++++---- .../0001-generated-python-name-collisions.md | 34 +++++++++++++++++++ docs/upstream-asks.md | 19 ++++++----- fixtures/Exhaustive.dhall | 4 +-- src/Interpreters/Project.dhall | 11 ++---- tests/conftest.py | 2 +- 9 files changed, 89 insertions(+), 67 deletions(-) create mode 100644 docs/adr/0001-generated-python-name-collisions.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8a2322f..b1d03de 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -51,12 +51,9 @@ jobs: with: ref: ${{ inputs.ref || github.sha }} - # A single-artifact pgn generate peaks at ~6.5 GB RSS on this runner - # (memory goes to normalizing the generator closure, near-independent - # of the query count), which flirts with the 7.8 GiB RAM + 3 GiB stock - # swap and got the VM killed by the memory watchdog mid-suite; extra - # swap lets the GC spill instead. /swapfile is taken by the image's - # stock swap, so the extra file goes on the /mnt ephemeral disk. + # pgn generation has exceeded the hosted runner's stock memory on this + # suite. Keep extra swap as evaluator and GC headroom. /swapfile is taken + # by the image's stock swap, so the extra file goes on /mnt. - name: Add swap headroom for pgn generate shell: bash run: | @@ -77,8 +74,8 @@ jobs: set -euo pipefail version="$(mise x -- pgn --version)" - if [[ "$version" != "0.9.1" ]]; then - echo "::error::Expected pgn 0.9.1 (mise.toml pin), got '$version'." + if [[ "$version" != "0.12.0" ]]; then + echo "::error::Expected pgn 0.12.0 (mise.toml pin), got '$version'." exit 1 fi @@ -124,9 +121,8 @@ jobs: with: ref: ${{ inputs.ref || github.sha }} - # A plain, standard-Dhall evaluator cannot run this because the local - # Project.buildLookup uses Text/equal, a builtin from pgn's forked Dhall. - # The pinned action bundles that same fork. + # Use the pinned directory-tree action for the same reproducible contract + # evaluation path used by release validation. - name: Generate output from Dhall uses: nikita-volkov/dhall-directory-tree.github-action@60a18dc647d6daea805263ea0fed7bb8011f3bcd # v2 with: diff --git a/CHANGELOG.md b/CHANGELOG.md index e6420ef..2f0001a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,10 +14,10 @@ module-internal audits). `pgn` 0.11.0 dropped the `Text/equal` builtin these relied on to compare two runtime `Text` values, and there is no `Text/replace`-based way to reconstruct that decision. A colliding schema is - no longer caught at `pgn generate`; it now fails the generated package's - `basedpyright --strict` gate instead (`reportRedeclaration` or - `reportInvalidTypeForm`), pointing at the generated Python rather than the - originating SQL/schema. + no longer caught at `pgn generate`. Collisions that remain visible in the + generated tree generally fail the package's `basedpyright --strict` gate + (`reportRedeclaration` or `reportInvalidTypeForm`), but a module-path + collision can overwrite an earlier file before the checker sees it. [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn to guarantee unique custom-type identities at the source. @@ -44,10 +44,12 @@ - Resolved custom-type references by their contract-supplied `CustomTypeRef.index` (gen-contract v5) rather than by name comparison, so a reference carries a stable schema-qualified identity (`pgSchema`/`pgName`) - directly. Same-unqualified-name types across schemas therefore no longer - collapse. Natural project indexes provide deterministic custom-import - deduplication and ordering. Query, parameter, field, and private statement - names are collision-safe while SQL names remain unchanged. + directly. Same-unqualified-name types across schemas no longer collapse + during reference resolution. Generated class and module names still come + from the unqualified contract name, so their Python names must remain unique. + Natural project indexes provide deterministic custom-import deduplication and + ordering. Reserved helper and keyword conflicts are escaped while SQL names + remain unchanged. - Kept `onUnsupported: Fail | Skip`. `Fail` aborts generation with the nested report. `Skip` preserves warnings and computes survivorship in a single @@ -101,13 +103,3 @@ and wheel are GPL-3.0-or-later because they inline gen-sdk. The wheel includes the sha256-pinned GPL text and `wheel/NOTICE`; PyPI publication remains disabled behind its explicit gate. - -- Bumped gen-contract to v5.0.0 and gen-sdk to v3.0.0. `Scalar.Custom` now - carries a `CustomTypeRef` (name, pgSchema, pgName, index) instead of a bare - `Name`, and `Value` flattens its optional array wrapper into plain - `dimensionality`/`elementIsNullable` fields. - -- Bumped the pinned `pgn` tool to v0.12.0 (from v0.9.1), required for the - gen-contract v5.0.0 support above. `pgn` 0.11.0, a prerequisite, dropped the - `Text/equal`/`Text/length`/`Bool/equal` builtins from its embedded Dhall - evaluator to stay in line with the official Dhall spec. diff --git a/DESIGN.md b/DESIGN.md index 8788429..1496cb6 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -186,11 +186,13 @@ dropped `Text/equal` (pgn 0.11.0), which removed the only mechanism that could compare two runtime `Text` values to decide a collision or resolve a mapping's `source` against a query/type name; that decision is not reconstructible from `Text/replace` alone (see section 10). Instead, the generated package is held -to `basedpyright --strict` returning zero errors and zero warnings; a -duplicate dataclass field name or a name that shadows a needed type surfaces -there (`reportRedeclaration`/`reportInvalidTypeForm`) rather than at `pgn -generate` time, and less precisely attributed (generated Python, not the -originating SQL/schema). +to `basedpyright --strict` returning zero errors and zero warnings. A duplicate +dataclass field name or a name that shadows a needed type surfaces there +(`reportRedeclaration`/`reportInvalidTypeForm`) rather than at `pgn generate` +time, and less precisely attributed (generated Python, not the originating +SQL/schema). A module-path collision can overwrite an earlier generated file +before the type checker runs, so input schemas must still guarantee unique +generated Python names. [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn to guarantee unique custom-type identities at the source. @@ -354,16 +356,17 @@ collapsing to the same unqualified contract `Name`), and project-wide facade/module audit, and per-type module-internal bindings). [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn to guarantee unique custom-type identities at the source instead. The -generated package's `basedpyright --strict` gate (see section 12) is now the -only backstop against a Python name collision reaching a consumer. +generated package's `basedpyright --strict` gate (see section 12) catches +collisions that remain visible in the output tree. It cannot detect an earlier +file that was overwritten at the same generated module path. -`buildLookup` — the last `Text/equal` user, which compared a reference's -snake-case name against each project custom type's name — has been removed. +`buildLookup`, the last `Text/equal` user, compared a reference's snake-case +name against each project custom type's name and has been removed. gen-contract v5's `CustomTypeRef` carries an `index` into `Project.customTypes`, and gen-sdk v3's `CustomTypes` module folds over that topologically-sorted list to compute survivorship without any text comparison. References now resolve by index (`Structures/CustomKind.dhall`'s `at`), so a repo-wide `grep -rn -"Text/equal" src/` finds only these explanatory comments — no live use remains. +"Text/equal" src/` finds only these explanatory comments. No live use remains. The generator no longer needs fork-only text equality anywhere. It does still rely on gen-contract v5's contract guarantees: every `CustomTypeRef.index` must diff --git a/README.md b/README.md index bec88da..258ac26 100644 --- a/README.md +++ b/README.md @@ -62,13 +62,16 @@ Query and custom-type identifiers are derived directly from their PostgreSQL source names (sanitized for Python syntax and reserved words only); a schema that maps two different SQL entities onto the same Python identifier will not be caught by `pgn generate`. Instead, the generated package is held to -`basedpyright --strict` with zero errors and zero warnings; a genuine -collision typically surfaces there as `reportRedeclaration` or -`reportInvalidTypeForm`, pointing at the generated Python rather than the -original SQL/schema source. Renaming the conflicting SQL or schema entity is -the fix. [pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) -asks pgn to guarantee unique custom-type identities at the source, which would -let a future version of this generator catch such collisions earlier again. +`basedpyright --strict` with zero errors and zero warnings. Collisions that +remain visible in the generated tree typically surface there as +`reportRedeclaration` or `reportInvalidTypeForm`, pointing at the generated +Python rather than the original SQL/schema source. That gate cannot detect a +module-path collision after one generated file has overwritten another, so +Python-name uniqueness remains an input-schema requirement. Renaming the +conflicting SQL or schema entity is the fix. +[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn +to guarantee unique custom-type identities at the source, which would let a +future version of this generator catch such collisions earlier again. ## Generated package diff --git a/docs/adr/0001-generated-python-name-collisions.md b/docs/adr/0001-generated-python-name-collisions.md new file mode 100644 index 0000000..d3024ee --- /dev/null +++ b/docs/adr/0001-generated-python-name-collisions.md @@ -0,0 +1,34 @@ +# ADR 0001: Generated Python name collisions + +Status: Superseded + +## Context + +Different PostgreSQL names can resolve to the same Python name. The original +implementation collected every generated binding and compared it with a growing +list of prior bindings. It did this for project exports and modules, query +members, custom-type members, and imports inside each custom-type module. + +Those scans were quadratic and expensive under Dhall normalization. They ran on +every generation, including projects that did not configure any name mappings. + +## Superseding decision + +The namespace-wide and local audits are removed. The typed query and custom-type +rename-mapping API and its runtime identity validators are removed with them. + +Lexical handling remains. Source-derived defaults are escaped for Python +keywords and generator-owned implementation names. + +Final uniqueness is now the project's responsibility. Two distinct entities can +resolve to the same module, class, function, field, or export. Static analysis +can catch many collisions in the surviving generated Python, but a module-path +collision can overwrite an earlier file before static analysis sees it. + +## Consequences + +- Normal generation uses less time and peak memory. +- Projects must keep final generated names unique. +- Resolve a conflict by renaming its SQL or schema source. +- A future collision check should avoid growing-list normalization and should + preferably be enforced by pgn before generator evaluation. diff --git a/docs/upstream-asks.md b/docs/upstream-asks.md index 1755509..1bac627 100644 --- a/docs/upstream-asks.md +++ b/docs/upstream-asks.md @@ -1,9 +1,10 @@ # Upstream resolution status: pgn and gen-sdk This brief records the current status of three upstream integration points. -The pgn annotation-metadata and warning-surfacing issues are closed and shipped. -Only the custom-type identity request remains actionable. Architecture details -stay in `DESIGN.md`. +The first two issues are closed and shipped. The qualified custom-type identity +gap is addressed by gen-contract v5, while the broader normalized-name +guarantee remains open in pgenie issue 75. Architecture details stay in +`DESIGN.md`. ## 1. pgn annotation metadata: resolved @@ -24,7 +25,7 @@ With `onUnsupported: Skip`, the generator retains a report for every dropped unit while a single-pass survivor fold removes unsupported custom types, their dependents, and affected statements. See `DESIGN.md`, section 8. -## 3. Preserve qualified custom-type identity: actionable +## 3. Preserve qualified custom-type identity: contract support shipped Project-wide custom-type lookup classifies every custom reference as an enum, composite, or absent. Shipped models are pure declarations with no class @@ -37,7 +38,9 @@ history: `Interpreters/Project.dhall` formerly implemented `buildLookup` by comparing a custom reference's snake-case name with each project custom type through `Text/equal`, which was the generator's sole need for that pgn-specific builtin; the unqualified comparison also exposed an upstream identity gap. That -lookup has been removed — references now resolve by `CustomTypeRef.index`. +lookup has been removed, references now resolve by `CustomTypeRef.index`. +Generated Python class and module names still use the unqualified contract +name, so gen-contract v5 does not itself prevent normalized-name collisions. Before gen-contract v5, a project containing `alpha.status` and `beta.status` could arrive with only one `customTypes` entry, while both uses were represented @@ -55,6 +58,6 @@ not enough. The qualified reference lets lookup use an equality-free structural match and removes the local `Text/equal` dependency described in `DESIGN.md`, section 10. -Changing `Scalar.Custom` affects gen-sdk consumers. This repository pins its -gen-sdk import by sha256, so adopting such a change would require an explicit -pin update and a full harness run. +Changing `Scalar.Custom` affects gen-sdk consumers. This repository adopted the +new shape through explicit sha256 pin updates to gen-contract v5 and gen-sdk v3 +and verified it with the full harness. diff --git a/fixtures/Exhaustive.dhall b/fixtures/Exhaustive.dhall index 5b5f2e6..fd1030b 100644 --- a/fixtures/Exhaustive.dhall +++ b/fixtures/Exhaustive.dhall @@ -9,9 +9,7 @@ -- those statements/types are dropped with a warning instead of aborting the -- whole compile. -- --- CI evaluates this fixture with the pinned fork-aware directory-tree action. --- Standard `dhall to-directory-tree` cannot evaluate the pgn fork's Text/equal --- builtin used by the generator. +-- CI evaluates this fixture with its pinned directory-tree action. let Sdk = ../src/Deps/Sdk.dhall let Gen = ../src/package.dhall diff --git a/src/Interpreters/Project.dhall b/src/Interpreters/Project.dhall index ddee955..68b62f4 100644 --- a/src/Interpreters/Project.dhall +++ b/src/Interpreters/Project.dhall @@ -76,10 +76,7 @@ let IndexedCustomType = { index : Natural, value : Model.CustomType } let LookupKind = < Composite | Enum | Domain > let LookupEntry = - { contractName : Text - , kind : LookupKind - , identity : CustomKind.Identity - } + { kind : LookupKind, identity : CustomKind.Identity } let ResolvedCustomType = { value : Model.CustomType, lookupEntry : LookupEntry } @@ -113,11 +110,7 @@ let resolveCustomTypes = customType.definition in { value = customType - , lookupEntry = - { contractName = customType.name.inSnakeCase - , kind - , identity - } + , lookupEntry = { kind, identity } } ) (Prelude.List.indexed Model.CustomType customTypes) diff --git a/tests/conftest.py b/tests/conftest.py index f276560..78c3bfd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -43,7 +43,7 @@ def pgn_bin() -> str: out = subprocess.run(["mise", "which", "pgn"], cwd=HERE, capture_output=True, text=True) path = out.stdout.strip() if out.returncode != 0 or not path: - pytest.fail("pgn 0.9.1 is required but is not resolvable via PATH or mise") + pytest.fail("pgn 0.12.0 is required but is not resolvable via PATH or mise") return path