GFQL: engine-dependent result shape forces per-engine branches at call sites (alias-marker column order, pivot dtype rules, marker residue)

[lmeyerov]

Problem

Several GFQL result-shape decisions are engine-dependent by accident of implementation, not by design. Because the divergences are observable in results, any code that produces a result without going through a given engine's canonical path has to reproduce that engine's quirk — so call sites grow if polars: ... elif cudf: ... else: ... branches, and every new fast path has to rediscover each quirk (usually via a failing parity test, sometimes only on a lane that isn't in the default selector).

Surfaced while landing #1776 / #1777, where three separate parity bugs all had this same root cause.

Concrete divergences found

1. Alias-marker column ORDER differs by engine. The canonical traversal adds boolean alias flag columns to the edge frame; pandas/cuDF put them first, native polars appends them. A path that skips the canonical traversal must re-synthesize them in each engine's order:

graphistry/compute/gfql/row/frame_ops.py:82-102 — an explicit _is_polars(...) branch whose entire purpose is column ordering.

2. The rows-pivot dtype rule differs by engine. pandas' pivot upcasts non-id int -> float64 and bool -> object; cuDF's preserves source dtypes. A lean projection that bypasses the pivot has to know which rule applies:

graphistry/compute/gfql_fast_paths.py:2201-2216 — is_cudf_rows gates whether the pandas artifact is applied. This one shipped as a silent wrong dtype on cuDF for as long as the fast path existed, because the cuDF lane is excluded from the default test selector.

3. Alias-marker residue exists on some engines and not others. pandas/cuDF leave a boolean flag column named after the alias in the frame; native polars deliberately omits it:

graphistry/compute/gfql/index/bindings.py:159-163 (_with_marker is a no-op on polars).

4. Engine-shaped plumbing that call sites keep re-deriving: filter routing (filter_by_dict vs filter_by_dict_polars), row-index semantics (reset_index is meaningless on polars), empty-frame construction, and join/concat kwargs. graphistry/compute/dataframe/join.py and graphistry/Engine.py already absorb some of this (df_concat's polars twin accepts and ignores ignore_index precisely so call sites don't branch) — that pattern works and should be the norm.

Why it matters

• Each divergence is a latent silent-wrong for any new fast path, and the failure mode is a dtype or column-order mismatch that only a strict differential test catches.
• The cuDF and polars-gpu lanes are not in the default selector, so a divergence can sit undetected for a long time (Don't columns to rename them, use bindings to refer to them instead #2 did).
• It pushes engine conditionals into semantic code, which is exactly where the vectorization/engine-compat guidance says they should not be.

Proposal

1. Standardize where the difference is arbitrary. Alias-marker column order and marker-residue presence look like implementation accidents rather than intentional engine semantics. Pick one canonical answer, make every engine produce it, and delete the branches. (Divergence Replace NaNs with nulls since node cannot parse JSON with NaNs #1 and Document API with pydoc #3 both look like this.)
2. Where a difference is genuinely forced (e.g. pandas' pivot upcast is pandas semantics, and polars genuinely has no row index), hide it behind a generic result-shape adapter — the Engine.df_concat / df_cons / df_to_engine pattern extended to result shaping: attach_alias_markers(frame, aliases, engine), pivot_result_dtypes(frame, cols, engine), empty_like(frame). Call sites then ask for the canonical shape and never name an engine.
3. Pin the contract with tests, not comments. A small cross-engine conformance test that asserts the shape contract (column order, marker presence, dtype rule) directly, parametrized over pandas/polars/cuDF (+ polars-gpu where reachable), so a new engine or a new fast path fails loudly instead of silently diverging.
4. Run the cuDF lane on GFQL result-shape changes — the divergence that shipped wrong was invisible to the default selector.

Happy to take this on as a follow-up; the three sites above are the concrete starting set, and #1776/#1777 already carry the differential tests that would prove any standardization keeps results identical.

🤖 Generated with Claude Code