Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 7 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,9 @@ jobs:
with:
ref: ${{ inputs.ref || github.sha }}

# A single-artifact pgn generate peaks at ~6.5 GB RSS on this runner
# (memory goes to normalizing the generator closure, near-independent
# of the query count), which flirts with the 7.8 GiB RAM + 3 GiB stock
# swap and got the VM killed by the memory watchdog mid-suite; extra
# swap lets the GC spill instead. /swapfile is taken by the image's
# stock swap, so the extra file goes on the /mnt ephemeral disk.
# pgn generation has exceeded the hosted runner's stock memory on this
# suite. Keep extra swap as evaluator and GC headroom. /swapfile is taken
# by the image's stock swap, so the extra file goes on /mnt.
- name: Add swap headroom for pgn generate
shell: bash
run: |
Expand All @@ -77,8 +74,8 @@ jobs:
set -euo pipefail

version="$(mise x -- pgn --version)"
if [[ "$version" != "0.9.1" ]]; then
echo "::error::Expected pgn 0.9.1 (mise.toml pin), got '$version'."
if [[ "$version" != "0.12.0" ]]; then
echo "::error::Expected pgn 0.12.0 (mise.toml pin), got '$version'."
exit 1
fi

Expand Down Expand Up @@ -124,9 +121,8 @@ jobs:
with:
ref: ${{ inputs.ref || github.sha }}

# A plain, standard-Dhall evaluator cannot run this because the local
# Project.buildLookup uses Text/equal, a builtin from pgn's forked Dhall.
# The pinned action bundles that same fork.
# Use the pinned directory-tree action for the same reproducible contract
# evaluation path used by release validation.
- name: Generate output from Dhall
uses: nikita-volkov/dhall-directory-tree.github-action@60a18dc647d6daea805263ea0fed7bb8011f3bcd # v2
with:
Expand Down
6 changes: 6 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,9 @@ already says.
`src/` pins its remote imports by sha256 (`src/Deps/*.dhall`). Bump those
deliberately, one at a time, and re-run the harness before committing a pin
change.

Never replace a pin with a local filesystem path (`../gen-contract/...`,
`../gen-sdk/...`) even temporarily for convenience; it only works on a
machine with those sibling repos checked out and breaks CI. Always use a
`https://raw.githubusercontent.com/pgenie-io/<repo>/<tag>/...` import with
its `sha256`.
67 changes: 40 additions & 27 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
# Upcoming

- Added fail-loud, namespace-wide validation for generated Python names. Query
and custom-type collisions can be resolved with typed, whole-entity mappings;
invalid, reserved, duplicate, and unknown mappings are rejected. Defensive
local-member audits identify both sources when such names reach the generator;
pgn may reject them earlier, and resolution is a SQL or schema rename. The
generator never silently overwrites files or assigns numeric suffixes.
Per-type module audits also reject a mapped class that would shadow an actual
primitive, core, or custom dependency import.
- Bumped the pins to gen-contract v5.0.0 and gen-sdk v3.0.0 (requires `pgn`
v0.12.0). Replaced the hand-rolled `buildLookup` custom-kind resolver and its
fixed-point removal cascade with gen-sdk's `CustomTypes` module: references
now resolve by the contract's `CustomTypeRef.index` and `onUnsupported: Skip`
computes survivorship in a single left-fold over the topologically-sorted
`customTypes`. This retires the generator's last `Text/equal` use, so no live
`Text/equal` call remains anywhere in `src/`.

- Removed the `queryNameMappings`/`customTypeNameMappings` rename-mapping
config and all generation-time Python namespace-collision detection
(project-wide facade/module, per-query, per-custom-type, and per-type
module-internal audits). `pgn` 0.11.0 dropped the `Text/equal` builtin these
relied on to compare two runtime `Text` values, and there is no
`Text/replace`-based way to reconstruct that decision. A colliding schema is
no longer caught at `pgn generate`. Collisions that remain visible in the
generated tree generally fail the package's `basedpyright --strict` gate
(`reportRedeclaration` or `reportInvalidTypeForm`), but a module-path
collision can overwrite an earlier file before the checker sees it.
[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn
to guarantee unique custom-type identities at the source.

- Finalized one additive package surface. Async functions remain at the package
root for every configuration. `emitSync: true` adds `<package>.sync`, a sync
Expand All @@ -29,22 +41,22 @@
ranks, composite ranks, custom-array fields inside composites, and missing or
unsupported custom types fail loudly.

- Retained `buildLookup` as the project-wide custom-kind resolver for identities
preserved in the contract. Its `Text/equal` use is an explicit pgn-fork
constraint. The planned exit must preserve every schema-qualified custom type
entry and expose a stable qualified identifier on `Scalar.Custom`; kind alone
is insufficient. pgn 0.9.1 can collapse same-unqualified-name types across
schemas, so that database shape remains unsupported and cannot be repaired by
a Python mapping. Natural project indexes now provide deterministic
custom-import deduplication and ordering.
Query, parameter, field, and private statement names are collision-safe while
SQL names remain unchanged.
- Resolved custom-type references by their contract-supplied
`CustomTypeRef.index` (gen-contract v5) rather than by name comparison, so a
reference carries a stable schema-qualified identity (`pgSchema`/`pgName`)
directly. Same-unqualified-name types across schemas no longer collapse
during reference resolution. Generated class and module names still come
from the unqualified contract name, so their Python names must remain unique.
Natural project indexes provide deterministic custom-import deduplication and
ordering. Reserved helper and keyword conflicts are escaped while SQL names
remain unchanged.

- Kept `onUnsupported: Fail | Skip`. `Fail` aborts generation with the nested
report. `Skip` preserves warnings and computes a bounded fixed point that
removes an unsupported custom type, its dependent custom types and statements,
and all affected type, registration, facade, Row, and statement entries. The
surviving package remains strict-importable.
report. `Skip` preserves warnings and computes survivorship in a single
left-fold over the topologically-sorted `customTypes`, removing an unsupported
custom type, its dependent custom types and statements, and all affected type,
registration, facade, Row, and statement entries. The surviving package
remains strict-importable.

- Established the generated tree as a greenfield layout with no compatibility
layer or promise for internal generated paths. Every generated Python file now
Expand All @@ -61,8 +73,8 @@
class-aware adapters, pure models, canonical `args_row` Rows, and no query
decoders, model codecs, casts, or bytes SQL. H2 CONFIRM: 25 Python files /
1467 lines / 11 statement files / 801 statement lines / 11 SQL. H3 CONFIRM:
Ruff 0/0, authored long0, SQL long0, raw output/no postformat. Tests: 73
passed, 0 skipped; pgn 0.9.1; strict basedpyright 0/0.
Ruff 0/0, authored long0, SQL long0, raw output/no postformat. Tests: 50
passed, 0 skipped; pgn v0.12.0; strict basedpyright 0/0.

- Migrated the generator to gen-contract v4.0.1 and gen-sdk v2.0.0 using
`Sdk.Sigs`. The implementation moved from `gen/` into `src/`, the working-tree
Expand All @@ -79,9 +91,10 @@
tuples with `dataclasses.fields` and `getattr`, preserving one-field arity.
The fixture exercises the type as both a parameter and a result.

- Kept `PyIdent.dhall` independent of fork-only text equality by using its
bounded `Text/replace` marker construction. Keyword fields and parameters gain
a trailing underscore only in Python, while raw database names remain stable.
- Kept `PyIdent.dhall` independent of fork-only text equality by using
`Lude.Text.replaceIfOneOf`'s bounded, sentinel-wrapped `Text/replace`
construction. Keyword fields and parameters gain a trailing underscore only
in Python, while raw database names remain stable.

- Preserved the release and license flow. The release job resolves
`src/package.dhall` into the `resolved.dhall` release asset, then byte-checks
Expand Down
145 changes: 80 additions & 65 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,22 +157,14 @@ directly. Statement SQL is passed as `LiteralString`. No runtime accepts encoded
SQL bytes. The generated package has no separately installed support library;
its only non-stdlib consumer dependency is `psycopg>=3.3.4,<4`.

## 5. Configuration, mapping, and unsupported shapes
## 5. Configuration and unsupported shapes

The public Dhall config is:

```dhall
{ packageName : Optional Text
, emitSync : Optional Bool
, onUnsupported : Optional < Fail | Skip >
, queryNameMappings : Optional
(List { source : Text, target : { snakeCase : Text, pascalCase : Text } })
, customTypeNameMappings : Optional
( List
{ source : { schema : Text, name : Text }
, target : { snakeCase : Text, pascalCase : Text }
}
)
}
```

Expand All @@ -182,35 +174,27 @@ The public Dhall config is:
- `packageName`: project name in kebab case;
- `emitSync`: `False`;
- `onUnsupported`: `Fail`.
- `queryNameMappings`: `[]`;
- `customTypeNameMappings`: `[]`.

An omitted config block, an omitted field, and a `null` field therefore use the
same fallback. Async output is present for every value of `emitSync`; only true
adds sync output.

Source-derived identifiers first receive lexical and reserved-name escaping.
Typed mappings replace a complete query or custom-type Python identity and must
already contain exact valid targets. After resolution, project and local
namespace audits reject duplicate modules, functions, Row classes, custom
types, facade exports, parameters, fields, and enum members before rendering.
Each custom-type module also audits its class against the primitive, core, and
custom dependency symbols it actually imports.
Fixed core exports remain occupied. The generator never silently overwrites a
file or assigns an order-dependent numeric suffix; see
[ADR 0001](docs/adr/0001-generated-python-name-collisions.md).

Custom-type mapping is exact only for entities preserved in the input contract.
pgn 0.9.1 can collapse same-unqualified-name types across schemas and expose an
unqualified `Scalar.Custom` reference. The generator cannot reconstruct the
discarded schema identity from SQL and rejects duplicate unqualified contract
names if they do reach it. Until upstream preserves all schema-qualified types
and a stable qualified reference, such database shapes are unsupported.

Local member audits are defensive for inputs that reach the generator. pgn may
reject a conflicting SQL or schema spelling during analysis first. Local
conflicts are resolved by renaming that SQL or schema source; whole-entity query
and custom-type mappings do not apply to members.
Source-derived identifiers receive lexical and reserved-name escaping only
(`Structures/PyIdent.dhall`); there is no rename-mapping config and no
generation-time namespace-collision detection. `pgn`'s embedded Dhall evaluator
dropped `Text/equal` (pgn 0.11.0), which removed the only mechanism that could
compare two runtime `Text` values to decide a collision or resolve a mapping's
`source` against a query/type name; that decision is not reconstructible from
`Text/replace` alone (see section 10). Instead, the generated package is held
to `basedpyright --strict` returning zero errors and zero warnings. A duplicate
dataclass field name or a name that shadows a needed type surfaces there
(`reportRedeclaration`/`reportInvalidTypeForm`) rather than at `pgn generate`
time, and less precisely attributed (generated Python, not the originating
SQL/schema). A module-path collision can overwrite an earlier generated file
before the type checker runs, so input schemas must still guarantee unique
generated Python names.
[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn
to guarantee unique custom-type identities at the source.

`Interpreters/Primitive.dhall` maps pgn union constructors, not signature-file
strings. Supported scalars are Boolean, integer and OID, floating point,
Expand Down Expand Up @@ -275,25 +259,31 @@ postprocessor rewrites generated files.
```text
optional config
-> resolved config
-> buildLookup(custom types)
-> fixed-point custom-type filtering in Skip mode
-> compile custom types and query checks
-> index-aligned kind classification (kindOf) of every custom type
-> one-pass survivor cascade (Sdk.CustomTypes.supportedCustomTypesReasoned)
masking unsupported indices to Absent in Skip mode
-> compile custom types and query checks against the masked lookup
-> dependency-first registration order
-> render facades, core, runtimes, types, registration, and statements
-> prepend the generated header
```

`onUnsupported: Fail` traverses the original project and propagates the first
compiled failure, so generation aborts without a partial client.

`onUnsupported: Skip` repeatedly rebuilds `buildLookup` from the surviving
custom types and removes any type that no longer compiles. It then compiles
queries against the final lookup. A removed type therefore also removes its
dependent composites and statements. Facade exports, type initializers,
registration entries, statement files, and Row exports are assembled only from
survivors. Reports for dropped units are preserved as warnings. The bounded
fixed point can only remove candidates, so it terminates after at most the
original custom-type count.
compiled failure, so generation aborts without a partial client. In this mode
nothing is masked: the lookup is the full, unmasked kind classification.

`onUnsupported: Skip` computes survivorship in a single left-fold via
`Sdk.CustomTypes.supportedCustomTypesReasoned`. Because `Project.customTypes`
is topologically sorted (every referenced index precedes the referencing type),
one pass suffices: the fold marks a type unsupported if its own definition
cannot compile (`ownDefinitionSupported`) or if any custom type it references
was already marked unsupported. Unsupported indices are then masked to `Absent`
in the kind lookup, so a reference to a removed type reads exactly like a
missing type. Queries compile against that same masked lookup, so a removed type
also removes its dependent composites and statements. Facade exports, type
initializers, registration entries, statement files, and Row exports are
assembled only from survivors, and reports for dropped units are preserved as
warnings.

Query compilation is deliberately consolidated in `QueryCheck`: keep/drop state
and warning data come from one literal `QueryGen.run` call site before the final
Expand Down Expand Up @@ -344,27 +334,52 @@ element nullability. `Member` and `ParamsMember` apply member nullability,
identifier safety, lookup classification, imports, and parameter wrapping.
`ResultColumns` builds Row field declarations in analyzed order. `Query`
combines SQL, Row, parameters, and surfaces. `Project` owns lookup scope,
fixed-point filtering, registration order, file selection, and facades.
single-pass survivor filtering, registration order, file selection, and
facades.

## 10. The pinned `Text/equal` constraint

`buildLookup` intentionally remains in `Interpreters/Project.dhall`. It compares
a custom reference's snake-case name with the project custom type name using
`Text/equal`. That builtin belongs to pgn's embedded Dhall fork and is not
available in upstream standard Dhall.

The dependency is pinned and explicit. The complete fixture needs fork-aware
evaluation solely because it invokes this generator and the local `buildLookup`
uses `Text/equal`. The upstream exit must preserve every schema-qualified
`customTypes` entry and put a stable qualified identifier, or a project index
with equivalent identity, on each custom scalar reference. Besides removing
text equality, that prevents pgn 0.9.1 from collapsing same-unqualified-name
types across schemas. Until then, pgn and CI's pinned fork-aware action are the
supported evaluators, and cross-schema duplicate type names are unsupported.

`PyIdent.dhall` uses its separate `Text/replace` marker construction for keyword
membership. `ImportSet.dhall` uses natural project indexes for equality,
deduplication, and ordering. Neither substitutes for the project lookup.
`pgn` 0.11.0 removed `Text/equal` (along with `Text/length` and `Bool/equal`)
from its embedded Dhall evaluator, to stay in line with the official Dhall
spec. There is no way to reconstruct a `Bool` or a differently-typed decision
from comparing two arbitrary runtime `Text` values using only `Text/replace`;
`Text/replace`-based tricks (this repository's own former keyword-marker
trick, and gen-sdk's `Lude.Text.replaceIfEqual`/`replaceIfOneOf`) only ever
transform text, they cannot branch into a different type. Every piece of this
generator that made a decision this way is gone: the `queryNameMappings`/
`customTypeNameMappings` rename-mapping config
(`Structures/PythonNameMapping.dhall`, matched a mapping's `source` against a
runtime name), `validateCustomTypeIdentities` (rejected two custom types
collapsing to the same unqualified contract `Name`), and
`Structures/PythonNamespace.dhall`'s `validate` (4 call sites in
`Interpreters/Project.dhall`: per-query and per-custom-type local audits, the
project-wide facade/module audit, and per-type module-internal bindings).
[pgenie-io/pgenie#75](https://github.com/pgenie-io/pgenie/issues/75) asks pgn
to guarantee unique custom-type identities at the source instead. The
generated package's `basedpyright --strict` gate (see section 12) catches
collisions that remain visible in the output tree. It cannot detect an earlier
file that was overwritten at the same generated module path.

`buildLookup`, the last `Text/equal` user, compared a reference's snake-case
name against each project custom type's name and has been removed.
gen-contract v5's `CustomTypeRef` carries an `index` into `Project.customTypes`,
and gen-sdk v3's `CustomTypes` module folds over that topologically-sorted list
to compute survivorship without any text comparison. References now resolve by
index (`Structures/CustomKind.dhall`'s `at`), so a repo-wide `grep -rn
"Text/equal" src/` finds only these explanatory comments. No live use remains.

The generator no longer needs fork-only text equality anywhere. It does still
rely on gen-contract v5's contract guarantees: every `CustomTypeRef.index` must
address the intended `customTypes` entry, and `customTypes` must be
topologically sorted (every referenced index precedes the referencing type),
which is what makes the single-pass survivor cascade sound. `pgn` remains the
supported evaluator and generation driver; upholding those index/ordering
invariants is the producer's responsibility.

`PyIdent.dhall` uses `Lude.Text.replaceIfOneOf`'s bounded `Text/replace`
construction for keyword membership. `ImportSet.dhall` uses natural project
indexes for equality, deduplication, and ordering. Neither substitutes for the
project lookup.

## 11. Taking ownership

Expand Down Expand Up @@ -394,7 +409,7 @@ PostgreSQL with these verdicts:
- H2 CONFIRM: 25 Python files / 1467 lines / 11 statement files / 801 statement
lines / 11 SQL.
- H3 CONFIRM: Ruff 0/0, authored long0, SQL long0, raw output/no postformat.
- Tests: 73 passed, 0 skipped; pgn 0.9.1; strict basedpyright 0/0.
- Tests: 50 passed, 0 skipped; pgn v0.12.0; strict basedpyright 0/0.

The H1 round trip covers both connection surfaces and exact cross-facade
identities. It covers scalar enum, enum arrays including rank 2, scalar
Expand Down
Loading
Loading