[EXPERIMENTAL] perf(gfql): seeded-chain fast path — slices 1-2 shipped, slice 3 (lean combine) in design

CHANGELOG.md

@@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine <gfql/engines>` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator).
 
 ### Fixed
+- **GFQL seeded Cypher/chain queries no longer pay two per-call O(E) costs on a resident graph (typed keep-mask + edge-frame augmentation caches)**: profiling the official LDBC IS1 point query on polars found the linear-with-edge-count terms that made Cypher-surface latency GROW with scale: (1) the #1658 index path rebuilt its simple-equality edge keep-mask via a full column compare every seeded hop (`_build_edge_keep_mask` — the profiled `eq_str` O(E) scan), and (2) the polars chain executor rebuilt `with_row_index` on the edge frame every call — an O(E) copy whose fresh object identity ALSO defeated every downstream id-keyed cache. Both are now recycle-safe caches keyed on the resident frame (weakref.finalize eviction, the proven `_pl_nan_to_null` clean-cache pattern): one augmented frame per resident edge frame (stable identity for all downstream caches) and one keep-mask per (frame, engine, scalar edge_match). Profiled: both O(E) terms eliminated from the warm path (`eq_str`/`with_row_index` gone). Parity: 274 tests (index 4-engine + polars conformance) unchanged. Slice 2: the combine path's order-column stamping (`_LazyShim`) eagerly `with_row_index`-copied BOTH node and edge frames O(N)+O(E) on every call before entering the fused plan — now deferred into the lazy plan itself (identical row numbering, single collect); 283 tests.
 - **GFQL polars engine natively runs whole-entity aggregation Cypher (LDBC IC4 shape): HAS_<Label> destination disambiguation + entity identity-key resolution + `key_prefixes` group-by expansion**: three previously honest-NIE polars lowerings, each an exact pandas-parity port: (1) `binding_rows_polars` now applies pandas' `_gfql_disambiguate_has_edge_destination_nodes` candidate-domain rule (an UNLABELED next-node op after a `HAS_<Label>`-typed edge narrows to that label ONLY when candidate node ids collide across labels) instead of declining every labeled-destination pattern; (2) `alias.__gfql_node_id__` (the #1650 whole-entity identity key the aggregation lowering groups by) resolves to the bare `alias` id column (the polars bindings table intentionally omits pandas' join-residue columns — the bare alias column IS the identity key); (3) `group_by(key_prefixes=...)` expands every `<prefix>*` row-table column into the key set exactly like pandas (functionally dependent on the identity key — group sizes unchanged). Result: the official LDBC IC4 (new-topics) query — comma-MATCH, `WITH DISTINCT`, CASE projections, whole-entity `sum` aggregation, HAVING-style `WHERE` — runs natively on polars, parity-exact (harness-verified vs expected rows at SF0.1: ok, 127.8 ms). Regression test `test_polars_rows_entity_groupby.py` (pandas oracle + polars parity on a discriminating fixture). Remaining polars `rows` declines are undirected bounded var-length (IC6/IC11) and shortestPath/zero-hop-unbounded (by design).
 - **GFQL connected OPTIONAL MATCH no longer crashes on polars frames (engine-polymorphic seed extraction)**: `_optional_arm_start_nodes` (gfql_unified.py) applied pandas-only ops (`.dropna()`/`.drop_duplicates()`/`.rename(columns=)`/boolean-mask indexing) to the joined binding rows, so an IS7-shaped `MATCH ... OPTIONAL MATCH ...` on `engine='polars'` crashed with `AttributeError: 'DataFrame' object has no attribute 'dropna'` before reaching the row pipeline (hit by LDBC SNB interactive-short-7 via the official query text). Now FIXED end-to-end: the seed extraction branches on frame type (`drop_nulls().unique().rename({...})` + `filter(is_in)` on polars), the optional-arm dispatch skips the pandas-only seeded-rows contract on polars (the seed restriction is a pruning hint — arm rows are left-outer-joined on the shared aliases, so unseeded is semantically identical, and `where`-arms already ran unseeded), and the arm join/alias-synthesis block gains a polars twin (`is_in` semi-join prune, `join(how='left')`, null-preserving alias synthesis). Result: **connected OPTIONAL MATCH now runs natively on polars** (previously pandas-only), oracle-exact on the IS7 shape; the simple-CASE null-literal equality (`CASE x WHEN null`, lowered as the `__cypher_case_eq__(x, null)` marker) now lowers natively to `is_null()` on polars (pandas-parity null-mask semantics; the general `CASE x WHEN v` form still declines honestly — it carries pandas' bool/numeric cross-dtype rules), so the full LDBC IS7 query INCLUDING its `CASE r WHEN null` flag projection runs natively on polars, oracle-exact. The polars arm is also PRUNED like the pandas one: since the polars bindings-row path declines `start_nodes` by contract, the same restriction is expressed as an id-membership filter injected into the arm's first (shared) node op — measured 9.2× on the LDBC SF0.1 harness (IS7 message-replies cypher 710.8→77.2 ms). pandas/cuDF paths byte-identical. Regression tests `test_optional_match_polars_frames.py` (pandas oracle, polars native end-to-end, polars no-pandas-ism contract).
 - **GFQL polars/polars-gpu seeded traversal no longer pays an O(E) NaN re-scan per hop on float-column graphs (identity-stable `_pl_nan_to_null` + clean-cache)**: every `hop()` runs `_coerce_input_formats` → `_pl_nan_to_null`, which scanned `is_nan().any()` over every float column on the (unchanged) resident edge frame **on every call** — so repeated seeded hops on a resident graph (the seeded-Search / native-hop pattern) grew O(E) with edge count on polars while pandas stayed flat. Measured: an indexed polars `g.hop` on 8M edges with one float column was **33.8 ms and growing**; it is now **0.20 ms and FLAT** (~140× faster; O(degree)), matching pandas. `_pl_nan_to_null` now probes an eager polars frame for real NaN once, returns a clean frame UNCHANGED (restoring the #1726 identity guard that #1731 reverted — so frame-identity caches like the #1658 index keep engaging), rewrites only the columns that genuinely carry NaN (values identical to the old unconditional `fill_nan`), and caches the id of frames verified clean so later calls skip the probe (recycle-safe via `weakref.finalize`; a rebound/new frame re-probes, so NaN→null semantics are unchanged). Regression: full 4-engine `test_index.py` parity + new `TestPlNanCleanCache` cases (NaN-present cleaned, clean-frame cached same-object, distinct frames independent).

graphistry/compute/filter_by_dict.py

@@ -82,15 +82,26 @@ def _label_series_contains(series: Any, label: str) -> Any:
     return series.apply(lambda value: label in _normalize_labels_cell(value))
 
 
+def _column_names(df: DataFrameT) -> Any:
+    """Column names without resolving a polars LazyFrame schema via ``.columns``, which warns
+    (PerformanceWarning) and re-resolves on every access — this runs per predicate per call on
+    the seeded fast path. ``collect_schema().names()`` is the sanctioned lazy form."""
+    collect_schema = getattr(df, "collect_schema", None)
+    if collect_schema is not None and type(df).__name__ == "LazyFrame":
+        return collect_schema().names()
+    return df.columns
+
+
 def resolve_filter_column(df: DataFrameT, col: str, val: Any) -> Tuple[str, Any]:
-    if col in df.columns:
+    names = _column_names(df)
+    if col in names:
         return col, val
 
     if col.startswith("label__") and val is True:
         label = col[len("label__") :]
-        if "labels" in df.columns:
+        if "labels" in names:
             return "labels", label
-        if "type" in df.columns and not _looks_like_edge_dataframe(df):
+        if "type" in names and not _looks_like_edge_dataframe(df):
             return "type", label
 
     from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError
@@ -100,7 +111,7 @@ def resolve_filter_column(df: DataFrameT, col: str, val: Any) -> Tuple[str, Any]
         f'Column "{col}" does not exist in dataframe',
         field=col,
         value=val,
-        suggestion=f'Available columns: {", ".join(df.columns[:10])}{"..." if len(df.columns) > 10 else ""}'
+        suggestion=f'Available columns: {", ".join(list(names)[:10])}{"..." if len(names) > 10 else ""}'
     )
 
 

graphistry/compute/gfql/index/traverse.py

@@ -14,7 +14,7 @@
 """
 from __future__ import annotations
 
-from typing import Any, List, Optional, Tuple, cast
+from typing import Any, Dict, List, Optional, Tuple, cast
 
 from typing_extensions import TypeGuard
 
@@ -74,6 +74,16 @@ def is_simple_equality_edge_match(
     return True
 
 
+# Recycle-safe cache of edge keep-masks: (id(edge frame), engine, scalar items) -> mask.
+# The mask is a pure function of the resident edge frame + a scalar edge_match, but was
+# rebuilt via a full O(E) column compare on EVERY seeded hop (profiled: the eq_str scan
+# is the linear term that makes typed Cypher point queries grow with edge count — ~2.8ms
+# per call at 1M rows). Same lifetime pattern as Engine._pl_nan_to_null's clean-cache:
+# weakref.finalize on the SOURCE frame evicts, so a recycled id can never serve a stale
+# mask while the original frame is alive.
+_EDGE_KEEP_MASK_CACHE: Dict[Tuple[int, str, tuple], "ArrayLike"] = {}
+
+
 def _build_edge_keep_mask(
     edges: DataFrameT, edge_match: EdgeMatch, engine: Engine, xp: "object"
 ) -> Optional[ArrayLike]:
@@ -82,12 +92,18 @@ def _build_edge_keep_mask(
     a simple-equality ``edge_match``.
 
     Built via each frame's native ``col == val`` (so cudf string columns stay on the
-    cudf layer instead of a cupy string compare). Returns ``None`` on ANY unexpected
-    shape or error, so the caller falls back to scan rather than risk a divergence.
+    cudf layer instead of a cupy string compare), then cached per resident frame —
+    repeated seeded hops on the same typed edges hit O(1). Returns ``None`` on ANY
+    unexpected shape or error, so the caller falls back to scan rather than risk a
+    divergence.
     """
     try:
         if not is_simple_equality_edge_match(edge_match):
             return None
+        cache_key = (id(edges), engine.value, tuple(sorted(edge_match.items())))
+        cached = _EDGE_KEEP_MASK_CACHE.get(cache_key)
+        if cached is not None:
+            return cached
         from graphistry.compute.filter_by_dict import (
             _is_numeric_dtype_safe, _is_string_dtype_safe,
         )
@@ -126,6 +142,13 @@ def _build_edge_keep_mask(
                 col_mask = cast(
                     ArrayLike, (series == val).fillna(False).to_numpy(dtype=bool))
             mask = col_mask if mask is None else cast(ArrayLike, cast(Any, mask) & cast(Any, col_mask))
+        if mask is not None:
+            try:
+                import weakref
+                weakref.finalize(edges, _EDGE_KEEP_MASK_CACHE.pop, cache_key, None)
+                _EDGE_KEEP_MASK_CACHE[cache_key] = mask
+            except TypeError:  # pragma: no cover - frame not weakref-able -> don't cache
+                pass
         return mask
     except Exception:  # pragma: no cover - defensive parity guard
         return None