Skip to content

DESIGN.md

Latest commit

 

History

History
435 lines (355 loc) · 19.3 KB

DESIGN.md

File metadata and controls

435 lines (355 loc) · 19.3 KB
Raw

python.gen design

Status: current. The Dhall under src/ and the live PostgreSQL harness under tests/ are the source of truth. This document describes the verified greenfield client, not an intermediate migration design.

The generator consumes pgn's analyzed project and emits one typed Python package. Async is always the root API. emitSync: true adds a sync facade and sync I/O functions without duplicating SQL or model definitions.

1. Ownership and package layout

pgn receives a List { path, content } from the generator and writes it below the artifact root. packageName is converted to an import name by replacing - with _.

For a project with custom types and emitSync: true, generated ownership is:

src/<pkg>/__init__.py
src/<pkg>/sync/__init__.py
src/<pkg>/_generated/__init__.py
src/<pkg>/_generated/_core.py
src/<pkg>/_generated/_runtime.py
src/<pkg>/_generated/_register.py
src/<pkg>/_generated/sync/_runtime.py
src/<pkg>/_generated/statements/__init__.py
src/<pkg>/_generated/statements/<query>.py
src/<pkg>/_generated/types/__init__.py
src/<pkg>/_generated/types/<custom_type>.py

The sync facade and runtime are omitted unless emitSync is true. Type and registration files are omitted when the project has no custom types. A consumer-owned shell, such as pyproject.toml and py.typed, sits around this output and is not generated.

There is one package, one generated type namespace, and one canonical statement module per query. Internal generated paths are greenfield implementation detail; the project makes no compatibility promise for them.

Interpreters/Project.dhall prepends one header to every Python file. Its exact first line is:

# @generated by python.gen (pGenie); regeneration overwrites manual changes.

The two SPDX lines immediately following it grant MIT-0 for emitted code. The marker is intentionally operational: manual edits are safe only after all regeneration paths have been disabled and the marker has been replaced.

2. Canonical statement modules and rows

Interpreters/Query.dhall compiles a query into a single statement module. The module contains, in order:

  • imports needed by parameters, results, and selected I/O surfaces;
  • one SQL string;
  • one @dataclass(frozen=True, slots=True) Row when the query returns rows;
  • the async function;
  • an adjacent function with the _sync suffix when emitSync is true.

The public facades alias those canonical objects. The root facade exports the async function. <pkg>.sync exports the adjacent sync function under the same public name. Both facades export the exact same Row classes, custom types, JsonValue, and NoRowError identities.

For a row-returning query the statement passes args_row(Row) to the runtime. psycopg's BaseRowFactory constructs the dataclass positionally in result-column order. There is no mapping row, generated query decoder, model codec, typing.cast, or second Row definition. This makes the statement's Row the canonical runtime result type for both surfaces.

Imports for custom annotations use the generated types namespace. The ImportSet project index is both a deduplication key and stable sort key, so a statement or composite imports each referenced type once in project order.

Python identifiers are sanitized only where Python syntax requires it. Parameters, result fields, composite fields, and query function names use the safe identifier. SQL placeholders and PostgreSQL names retain their original spelling. For example, a PostgreSQL column named class becomes the Python field class_, while its positional value still occupies the analyzed column position.

3. Adapter-owned custom types and registration

Generated models are pure declarations:

  • enums are StrEnum subclasses containing the exact PostgreSQL labels;
  • composites are frozen, slotted dataclasses containing their typed fields;
  • statement rows are frozen, slotted dataclasses.

They contain no database conversion methods. psycopg owns loading and dumping. Templates/RegisterModule.dhall emits class-aware registration into one shared module:

  • EnumInfo.fetch and register_enum receive the generated enum class and an explicit member-to-label mapping;
  • CompositeInfo.fetch and register_composite receive the generated dataclass plus validated object and sequence callbacks.

The callback factory compares sanitized PostgreSQL field names with dataclass field names before constructing or flattening an object. This catches schema and model drift at the adapter boundary.

register_types is async. When sync output is enabled, the same module also contains register_types_sync, which the sync facade exports as register_types. Registration is a per-new-connection precondition and must run after the database migrations and custom types exist. Missing catalog entries raise LookupError rather than falling back to text.

Registration order is dependency-first. Enums are ready immediately; composites are emitted only when all referenced custom types are already ready. A bounded pass either resolves the full order or reports unresolved or cyclic dependencies. This ordering lets psycopg recursively adapt nested scalar composites and arrays of the supported ranks.

Verified custom shapes are scalar enums, enum arrays through rank 2, scalar composite load and dump, rank-1 composite arrays, nested scalar composites, nullable fields, and sanitized fields. Async and sync return the same exact model identities.

4. Runtime and surfaces

Structures/Surface.dhall is a small render-token record for adjacent functions:

{ defKeyword : Text
, connType : Text
, awaitKw : Text
, functionSuffix : Text
, runtimePrefix : Text
, helperSuffix : Text
, corePrefix : Text
}

The async value renders async def, AsyncConnection, and await. The sync value renders def, Connection, and _sync implementation names. It does not select between mutually exclusive package shapes.

_core.py is emitted once and owns JsonValue and NoRowError. The async runtime is always emitted; the sync runtime is additive. Each runtime contains the same five cardinality helpers:

helper result
fetch_optional `Row
fetch_single Row, or NoRowError
fetch_many list[Row]
execute_rows_affected int
execute_void None

Row helpers accept BaseRowFactory[T] and return psycopg's constructed object directly. Statement SQL is passed as LiteralString. No runtime accepts encoded SQL bytes. The generated package has no separately installed support library; its only non-stdlib consumer dependency is psycopg>=3.3.4,<4.

5. Configuration and unsupported shapes

The public Dhall config is:

{ packageName : Optional Text
, emitSync : Optional Bool
, onUnsupported : Optional < Fail | Skip >
}

src/package.dhall passes that type and an all-None default to Sdk.Sigs.generator. Project.run resolves each field independently:

  • packageName: project name in kebab case;
  • emitSync: False;
  • onUnsupported: Fail.

An omitted config block, an omitted field, and a null field therefore use the same fallback. Async output is present for every value of emitSync; only true adds sync output.

Source-derived identifiers receive lexical and reserved-name escaping only (Structures/PyIdent.dhall); there is no rename-mapping config and no generation-time namespace-collision detection. pgn's embedded Dhall evaluator dropped Text/equal (pgn 0.11.0), which removed the only mechanism that could compare two runtime Text values to decide a collision or resolve a mapping's source against a query/type name; that decision is not reconstructible from Text/replace alone (see section 10). Instead, the generated package is held to basedpyright --strict returning zero errors and zero warnings. A duplicate dataclass field name or a name that shadows a needed type surfaces there (reportRedeclaration/reportInvalidTypeForm) rather than at pgn generate time, and less precisely attributed (generated Python, not the originating SQL/schema). A module-path collision can overwrite an earlier generated file before the type checker runs, so input schemas must still guarantee unique generated Python names. pgenie-io/pgenie#75 asks pgn to guarantee unique custom-type identities at the source.

Interpreters/Primitive.dhall maps pgn union constructors, not signature-file strings. Supported scalars are Boolean, integer and OID, floating point, numeric, textual, UUID, date/time/timestamp/interval, bytea, and JSON. Arrays wrap the scalar type once per analyzed dimension, preserving element and outer nullability independently. JSON parameters use psycopg's Json or Jsonb wrapper only for a scalar value.

CustomKind.Lookup classifies each custom reference as enum, composite, or absent. For enum and composite references it returns only the generated Python class and module identity plus stable project order; composite fields stay with the original model. That classification drives type imports, supported array ranks, nested dependencies, and adapter registration. The generator reports instead of guessing when a primitive or custom type cannot be mapped.

The explicit custom limits are:

  • enum dimensions above 2 are unsupported;
  • composite dimensions above 1 are unsupported;
  • a custom-array field inside a composite is unsupported;
  • an absent or unsupported custom type is unsupported.

Domains are unsupported. JSON array parameters are also rejected because a single JSON wrapper cannot adapt elements faithfully. Every failure carries the query or type path and member name through Compiled.nest.

6. Cardinality fidelity

Interpreters/Result.dhall preserves pgn's cardinality verbatim:

  • Rows Optional maps to Row | None and fetch_optional;
  • Rows Single maps to Row and fetch_single;
  • Rows Multiple maps to list[Row] and fetch_many;
  • RowsAffected maps to int and execute_rows_affected;
  • Void maps to None and execute_void.

The generator does not infer a narrower return from SQL predicates. If pgn classifies a returning statement as multiple, its API remains list[Row].

7. SQL rendering

Interpreters/QueryFragments.dhall converts pgn fragments into one Python triple-quoted SQL string in each statement module:

  • a variable becomes %(<raw parameter name>)s;
  • raw SQL remains verbatim after Python-literal escaping;
  • literal percent signs are doubled for psycopg named style;
  • the variable branch is not percent-doubled.

The literal begins with a backslash so the first SQL line is flush and retains a trailing newline. Parameter dictionaries use raw PostgreSQL names as keys and sanitized Python expressions as values. Source SQL owns explicit database casts; the generator does not append adaptation casts. The same SQL object is passed to async and sync helpers as a string whose static type satisfies LiteralString. Output is the raw Dhall render; no Python formatter or SQL postprocessor rewrites generated files.

8. Project pipeline and onUnsupported

Interpreters/Project.dhall is the project-level coordinator:

optional config
  -> resolved config
  -> index-aligned kind classification (kindOf) of every custom type
  -> one-pass survivor cascade (Sdk.CustomTypes.supportedCustomTypesReasoned)
     masking unsupported indices to Absent in Skip mode
  -> compile custom types and query checks against the masked lookup
  -> dependency-first registration order
  -> render facades, core, runtimes, types, registration, and statements
  -> prepend the generated header

onUnsupported: Fail traverses the original project and propagates the first compiled failure, so generation aborts without a partial client. In this mode nothing is masked: the lookup is the full, unmasked kind classification.

onUnsupported: Skip computes survivorship in a single left-fold via Sdk.CustomTypes.supportedCustomTypesReasoned. Because Project.customTypes is topologically sorted (every referenced index precedes the referencing type), one pass suffices: the fold marks a type unsupported if its own definition cannot compile (ownDefinitionSupported) or if any custom type it references was already marked unsupported. Unsupported indices are then masked to Absent in the kind lookup, so a reference to a removed type reads exactly like a missing type. Queries compile against that same masked lookup, so a removed type also removes its dependent composites and statements. Facade exports, type initializers, registration entries, statement files, and Row exports are assembled only from survivors, and reports for dropped units are preserved as warnings.

Query compilation is deliberately consolidated in QueryCheck: keep/drop state and warning data come from one literal QueryGen.run call site before the final render call. Dhall normalization substitutes call sites, and multiplying those sites caused a measured minutes-scale regression. Custom-type checks use their separately measured fast shape.

9. Generator decomposition

The source follows the gen-sdk interpreter/template split:

src/
  package.dhall
  Deps/                       # sha256-pinned remote packages
  Structures/
    CustomKind.dhall
    ImportSet.dhall
    OnUnsupported.dhall
    PyIdent.dhall
    Surface.dhall
  Interpreters/
    Primitive.dhall
    Scalar.dhall
    Value.dhall
    Member.dhall
    ParamsMember.dhall
    CustomType.dhall
    QueryFragments.dhall
    ResultColumns.dhall
    Result.dhall
    Query.dhall
    Project.dhall
  Templates/
    CoreModule.dhall
    RuntimeModule.dhall
    StatementModule.dhall
    RegisterModule.dhall
    FacadeModule.dhall
    EnumModule.dhall
    CompositeModule.dhall
    TypesInit.dhall
    InitModule.dhall

Scalar identifies primitive versus custom values. Value adds array rank and element nullability. Member and ParamsMember apply member nullability, identifier safety, lookup classification, imports, and parameter wrapping. ResultColumns builds Row field declarations in analyzed order. Query combines SQL, Row, parameters, and surfaces. Project owns lookup scope, single-pass survivor filtering, registration order, file selection, and facades.

10. The pinned Text/equal constraint

pgn 0.11.0 removed Text/equal (along with Text/length and Bool/equal) from its embedded Dhall evaluator, to stay in line with the official Dhall spec. There is no way to reconstruct a Bool or a differently-typed decision from comparing two arbitrary runtime Text values using only Text/replace; Text/replace-based tricks (this repository's own former keyword-marker trick, and gen-sdk's Lude.Text.replaceIfEqual/replaceIfOneOf) only ever transform text, they cannot branch into a different type. Every piece of this generator that made a decision this way is gone: the queryNameMappings/ customTypeNameMappings rename-mapping config (Structures/PythonNameMapping.dhall, matched a mapping's source against a runtime name), validateCustomTypeIdentities (rejected two custom types collapsing to the same unqualified contract Name), and Structures/PythonNamespace.dhall's validate (4 call sites in Interpreters/Project.dhall: per-query and per-custom-type local audits, the project-wide facade/module audit, and per-type module-internal bindings). pgenie-io/pgenie#75 asks pgn to guarantee unique custom-type identities at the source instead. The generated package's basedpyright --strict gate (see section 12) catches collisions that remain visible in the output tree. It cannot detect an earlier file that was overwritten at the same generated module path.

buildLookup, the last Text/equal user, compared a reference's snake-case name against each project custom type's name and has been removed. gen-contract v5's CustomTypeRef carries an index into Project.customTypes, and gen-sdk v3's CustomTypes module folds over that topologically-sorted list to compute survivorship without any text comparison. References now resolve by index (Structures/CustomKind.dhall's at), so a repo-wide grep -rn "Text/equal" src/ finds only these explanatory comments. No live use remains.

The generator no longer needs fork-only text equality anywhere. It does still rely on gen-contract v5's contract guarantees: every CustomTypeRef.index must address the intended customTypes entry, and customTypes must be topologically sorted (every referenced index precedes the referencing type), which is what makes the single-pass survivor cascade sound. pgn remains the supported evaluator and generation driver; upholding those index/ordering invariants is the producer's responsibility.

PyIdent.dhall uses Lude.Text.replaceIfOneOf's bounded Text/replace construction for keyword membership. ImportSet.dhall uses natural project indexes for equality, deduplication, and ordering. Neither substitutes for the project lookup.

11. Taking ownership

The generated marker means regeneration is authoritative. A consumer may eject only as a coordinated transition:

  1. generate and commit the intended complete client;
  2. disable every local and CI regeneration path before edits;
  3. replace the marker with a project ownership header;
  4. maintain the result as ordinary Python;
  5. keep the package structure unless intentionally refactoring all imports;
  6. gate the result with basedpyright strict, Ruff, and database tests.

Once regeneration is disabled, there is no generator behavior involved. Before that point, manual edits are expected to be overwritten.

12. Verification evidence

The final fixture was regenerated as raw output and verified against live PostgreSQL with these verdicts:

  • H1 CONFIRM: psycopg 3.3.4; consumer psycopg>=3.3.4,<4; class-aware EnumInfo/register_enum and CompositeInfo/register_composite; dependency-first registration; pure StrEnum and frozen slotted dataclasses; args_row constructs canonical Row; no query decoders, model codecs, casts, or bytes SQL.
  • H2 CONFIRM: 25 Python files / 1467 lines / 11 statement files / 801 statement lines / 11 SQL.
  • H3 CONFIRM: Ruff 0/0, authored long0, SQL long0, raw output/no postformat.
  • Tests: 50 passed, 0 skipped; pgn v0.12.0; strict basedpyright 0/0.

The H1 round trip covers both connection surfaces and exact cross-facade identities. It covers scalar enum, enum arrays including rank 2, scalar composite load and dump, rank-1 composite arrays, nested scalar composites, nullable fields, and sanitized names. Negative contract tests cover unsupported ranks, custom-array fields inside composites, and missing or unsupported custom types.

13. Development and release constraints

Use mise for every repository command. mise run test drives real pgn subprocesses against a live PostgreSQL server; PGN_TEST_DATABASE_URL selects a non-default server. The harness only creates and drops its own scratch database. Golden regeneration is intentionally serial and memory-heavy; use only mise run golden and review its diff.

src/Deps/*.dhall pins remote imports by sha256 and should be changed one at a time. src/package.dhall is the working-tree generator entry point. fixtures/Exhaustive.dhall is the contract fixture. The release workflow resolves the entry point into resolved.dhall, publishes that release asset, then byte-checks and bundles it into the wheel. Repository source is MIT, generated Python is MIT-0, and the combined resolved artifact and wheel are GPL-3.0-or-later because they inline gen-sdk.