perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal

CHANGELOG.md

@@ -8,6 +8,18 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ## [Development]
 <!-- Do Not Erase This Section - Used for tracking unreleased changes -->
 
+### Performance
+- **GFQL indexed fixed-hop bindings now dispatch BEFORE the canonical traversal (pandas / cuDF / native Polars)**: the resident-index fixed-hop path (`rows(binding_ops=...)`, the Cypher multi-alias `MATCH … RETURN` shape) was consulted only *inside* row materialization — after the engine had already run the full canonical traversal — so its compact path bag was computed on top of the graph-sized work it exists to avoid. Both the pandas/cuDF chain boundary and the native Polars chain now ask the same shared, structural gate first and, only when it can serve the WHOLE middle exactly, skip the canonical traversal and hand the compact state to the unchanged row materializer. Engagement stays operator/index/dtype/cost based — no query, schema, or hop-count recognition — and every unsupported, seeded, prefiltered, policy-bearing, shortest-path, or cost-gated shape still declines to canonical execution with identical results. On a standard LDBC SNB SF1 interactive-short profile this removed the redundant two-hop typed-mask and 16-merge buckets and cut the profiled call ~11.9× (1263 ms → 106 ms), exact 19-row oracle preserved.
+- **Indexed seed lookup now covers a unique node id PLUS extra scalar constraints**: `{node_id: v, other: w}` previously fell back to a full node-table scan even though the unique node-id index can gather at most one row; it now gathers that one row and applies the remaining constraints to it. Missing ids and non-matching extra constraints return the same empty result as before, and duplicate-id graphs (which cannot build the index) are unaffected.
+- **Seeded typed-hop property projection keeps its fast path through DISTINCT / ORDER BY / SKIP / LIMIT**: those trailing row ops are plain frame operations, so the fast path now delegates them to the canonical chain instead of declining the whole query.
+
+### Changed
+- **GFQL execution context is declared rather than attached at runtime**: the private `_gfql_*` per-execution fields (index policy and registry, row-pipeline base graph, carried seed nodes, edge aliases, shortest-path backend, and the indexed-bindings handoff) are now declared on `Plottable` with defaults on `PlotterBase`, instead of being set with `setattr` and read back with `getattr(..., default)`. No public API or behaviour change; internal call sites are typed, and hand-rolled `Plottable` stand-ins must now construct the full context.
+
+### Fixed
+- **Indexed bypass could return every node for `rows()` over an unnamed pattern**: the boundary accepted a `rows()` call carrying no binding ops over a middle with no aliases. That call reads the traversal-narrowed node table, but the bypass hands it the full graph, so the query would have returned all nodes. The gate now requires that the rows call actually consumes the whole middle as binding ops — either it already carries them, or the named-middle rewrite installs exactly them.
+- **Seeded property-RETURN dtype divergence on cuDF**: the lean projection applied the pandas rows-pivot artifact (int → float64, bool → object) on every engine, but cuDF's canonical pivot preserves the source dtypes — so the fast path returned `float64`/`object` where cuDF's own canonical path returns `int64`/`bool`. The cast rule is now engine-aware; the dtype-class decline guard is unchanged.
+
 ### Documentation
 - **GFQL pay-as-you-go resident indexing user guide**: New :doc:`Pay-As-You-Go Resident Indexing <gfql/indexing>` page — the lifecycle guide to resident indexes (`gfql_index_all()` / `gfql_index_edges()` / `create_index()` / `show_indexes()` / `drop_index()`): what the node-id + CSR in/out adjacency sidecars are, what engages them on 0.58.0 (seeded typed-hop fast paths incl. property RETURNs and property-seeded lookups per #1768/#1770, direct `g.hop()`; the general polars chain traversal honestly noted as not yet covered), the staleness/validity contract (identity + fingerprint; rebind invalidates; declines are safe — identical results either way), engine notes (polars needs `gfql_index_all(engine='polars')` until #1767), 0.58.0-tag measured numbers, and a runnable end-to-end example. Wired into the GFQL toctree + recommended paths alongside :doc:`Seeded Traversal Indexes <gfql/index_adjacency>`.
 - **GFQL performance docs: 0.58.0 release-tag-verified numbers, siloed in one page**: `gfql/performance.rst` is now the canonical benchmark-numbers page (alongside `gfql/index_adjacency.rst` for the index benchmarks) — a benchmark rerun updates it alone. It carries the 0.58.0 tag sweep (DGX Spark GB10, warm medians N=30; four-engine numbers cross-engine parity-verified, competitor pairs validated against expected result rows): seeded typed-hop fast path across all four engines (e.g. pandas 29.9→2.46ms, 12.1×), native chain form, resident-index covered-shape lookups (with the `gfql_index_all(engine='polars')` caveat / PR #1767), flat seeded-hop scaling on pandas (0.159–0.164ms from 0.25M to 32M edges), the one-keyword `engine='polars'` LDBC SNB SF1 seed-lookup win (1,299.6→106.1ms, 12.3×), LDBC SNB interactive SF1 vs Neo4j 5.26 same-box pairs (GFQL 4 of 5; Neo4j wins recent-replies — reported as-is), OLAP multi-join vs embedded Kuzu (q8 200×, q9 14.2×) with the honest inverse (Kuzu wins single-table aggregates 2–4×, seeded property-projection lookups 2.4–64×), plus the prior Orkut/LiveJournal bulk sweep (moved from `engines.rst`, dated once) and its methodology. All other pages — `engines.rst`, `quick.rst`, `about.rst`, `overview.rst`, `index.rst` — now carry stable qualitative claims (e.g. "often an order of magnitude faster on query-heavy workloads") that link into `performance.rst` instead of inline figures, replacing the stale "up to ~38×" headlines and avoiding scattered per-claim version labels.

agents/skills/review/SKILL.md

@@ -211,6 +211,20 @@ GFQL is vectorization-first and pure-functional. The row pipeline runs on pandas
 
 Vectorized mutation (`df.loc[mask, col] = v`) is still mutation — prefer `df.assign(...)`.
 
+**Batch the assign** (SUGGESTION; IMPORTANT on a hot row path): copy-then-mutate
+(`out = df.copy()` followed by one or more `out[col] = ...`) should be a single
+`df.assign(col_a=..., col_b=...)`. It is the functional phrasing the rest of the
+codebase uses, it drops the explicit copy (assign already returns a new frame), and
+one assign builds the result once instead of copying and then writing column by
+column.
+
+| Pattern | Fix |
+|---|---|
+| `out = df.copy()` + `out[c] = v` | `df.assign(c=v)` |
+| `out = df.copy()` + several `out[c] = v` | one `df.assign(c1=v1, c2=v2)` |
+| `df.copy().rename(columns=...)` | `df.rename(columns=...)` — rename already copies |
+| `out = df[mask].copy()` with no later mutation | `df[mask]` — masking already copies |
+
 **cuDF compatibility** — flag IMPORTANT unless noted:
 
 | Pattern | Fix |

bin/ci_cypher_surface_guard_baseline.json

@@ -13,5 +13,5 @@
       "max_properties": 0
     }
   },
-  "lowering_py_max_lines": 9249
+  "lowering_py_max_lines": 9255
 }

graphistry/Plottable.py

@@ -1,4 +1,4 @@
-from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Set, Tuple, Union, Protocol, overload
+from typing import TYPE_CHECKING, Any, Callable, Dict, Iterable, List, Optional, Set, Tuple, Union, Protocol, overload
 from typing_extensions import Literal, runtime_checkable
 import pandas as pd
 
@@ -22,6 +22,10 @@
 from graphistry.models.surfaces.graphistry_frontend.url_params import URLParamsDict
 
 if TYPE_CHECKING:
+    from graphistry.compute.typing import DataFrameT
+    from graphistry.compute.gfql.index.handoff import IndexedBindingsHandoff
+    from graphistry.compute.gfql.index.policy import IndexPolicy
+    from graphistry.compute.gfql.index.registry import GfqlIndexRegistry
     try:
         from umap import UMAP
     except:
@@ -56,6 +60,17 @@ class Plottable(Protocol):
     session: ClientSession
     _pygraphistry: AuthManagerProtocol
 
+    # GFQL execution context. Declared here (rather than smuggled onto instances
+    # with setattr) so every access is typed and the defaults live in one place —
+    # see PlotterBase for the values. Internal: not part of the user surface.
+    _gfql_index_policy: "IndexPolicy"
+    _gfql_index_registry: Optional["GfqlIndexRegistry"]
+    _gfql_indexed_bindings_handoff: Optional["IndexedBindingsHandoff"]
+    _gfql_rows_base_graph: Optional["Plottable"]
+    _gfql_start_nodes: Optional["DataFrameT"]
+    _gfql_rows_edge_aliases: Optional[Iterable[str]]
+    _gfql_shortest_path_backend: str
+
     _edges : Any
     _nodes : Any
     _source : Optional[str]

graphistry/PlotterBase.py

@@ -1,5 +1,11 @@
 from graphistry.Plottable import Plottable, RenderModes, RenderModesConcrete
-from typing import Any, Callable, Dict, List, Optional, Union, Tuple, cast, overload, TYPE_CHECKING
+from typing import Any, Callable, Dict, Iterable, List, Optional, Union, Tuple, cast, overload, TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from graphistry.compute.typing import DataFrameT
+    from graphistry.compute.gfql.index.handoff import IndexedBindingsHandoff
+    from graphistry.compute.gfql.index.policy import IndexPolicy
+    from graphistry.compute.gfql.index.registry import GfqlIndexRegistry
 from typing_extensions import Literal
 from graphistry.io.types import ComplexEncodingsDict
 from graphistry.models.collections import CollectionsInput
@@ -213,6 +219,18 @@ class PlotterBase(Plottable):
     The class supports convenience methods for mixing calls across Pandas, NetworkX, and IGraph.
     """
 
+    # GFQL execution context defaults (fields declared on Plottable). Immutable
+    # scalars / None only, so sharing the class attribute is safe; `bind()` is a
+    # shallow copy, so a per-graph value set on an instance travels with it.
+    _gfql_index_policy: "IndexPolicy" = "use"
+    _gfql_index_registry: Optional["GfqlIndexRegistry"] = None
+    _gfql_indexed_bindings_handoff: Optional["IndexedBindingsHandoff"] = None
+    _gfql_rows_base_graph: Optional["Plottable"] = None
+    _gfql_start_nodes: Optional["DataFrameT"] = None
+    _gfql_rows_edge_aliases: Optional[Iterable[str]] = None
+    _gfql_shortest_path_backend: str = "auto"
+    _g: Optional["Plottable"] = None  # set only on row-pipeline adapters
+
     _defaultNodeId = NODE
     _defaultEdgeSourceId = SRC
     _defaultEdgeDestinationId = DST

graphistry/compute/chain.py

@@ -8,7 +8,7 @@
 from graphistry.Engine import safe_merge
 from graphistry.util import setup_logger
 from graphistry.utils.json import JSONVal
-from .ast import ASTObject, ASTNode, ASTEdge, Direction, from_json as ASTObject_from_json, serialize_binding_ops
+from .ast import ASTObject, ASTNode, ASTEdge, ASTCall, Direction, from_json as ASTObject_from_json, serialize_binding_ops
 from .typing import DataFrameT, SeriesT
 from .util import generate_safe_column_name
 from .chain_fast_paths import _seeded_typed_hop_pandas_cudf
@@ -25,6 +25,7 @@
 
 if TYPE_CHECKING:
     from graphistry.compute.exceptions import GFQLSchemaError, GFQLValidationError
+    from graphistry.compute.gfql.index.handoff import IndexedBindingsHandoff
 
 logger = setup_logger(__name__)
 
@@ -556,6 +557,71 @@ def _get_boundary_calls(ops: List[ASTObject]) -> Tuple[List[ASTObject], List[AST
     return (prefix, middle, suffix)
 
 
+
+def _plan_indexed_middle(
+    g: Plottable,
+    prefix: List[ASTObject],
+    middle: List[ASTObject],
+    suffix: List[ASTObject],
+    engine: Union[EngineAbstract, str],
+    policy: Optional[Any],
+    start_nodes: Optional[DataFrameT],
+) -> Optional["IndexedBindingsHandoff"]:
+    """Decide ONCE whether the resident indexes can serve this boundary's middle.
+
+    Returns ``None`` when the indexed path does not apply to this shape at all, a
+    handoff carrying a state when it serves, and a handoff without one when it was
+    tried and declined. The twin of ``_try_indexed_middle_polars`` on the polars
+    chain; keeping both as a single predicate-plus-plan keeps the two boundaries
+    comparable instead of two inline condition chains that drift.
+    """
+    from .gfql.index.handoff import IndexedBindingsHandoff
+
+    if not (middle and suffix) or prefix or start_nodes is not None or policy:
+        return None
+    call = suffix[0]
+    if not isinstance(call, ASTCall) or call.function != "rows":
+        return None
+    if (
+        call.params.get("source") is not None
+        or call.params.get("alias_endpoints") is not None
+        or call.params.get("alias_prefilters")
+        or not all(isinstance(op, (ASTNode, ASTEdge)) for op in middle)
+    ):
+        return None
+    plan = serialize_binding_ops(middle)
+    # The bypass is only sound when the rows call actually consumes the WHOLE
+    # middle as binding ops: either it already carries them, or the named-middle
+    # rewrite installs exactly them. A rows call with no binding ops over an
+    # UNNAMED middle instead reads the traversal-narrowed node table, which the
+    # bypass would not produce.
+    if not (
+        call.params.get("binding_ops") == plan
+        or (
+            call.params.get("binding_ops") is None
+            and any(op._name is not None for op in middle
+                    if isinstance(op, (ASTNode, ASTEdge)))
+        )
+    ):
+        return None
+
+    engine_concrete = resolve_engine(engine, g)  # type: ignore[arg-type]
+    if engine_concrete not in (Engine.PANDAS, Engine.CUDF):
+        return None
+
+    from .gfql.index.bindings import try_indexed_connected_bindings_state
+
+    state = try_indexed_connected_bindings_state(g, middle, engine=engine_concrete)
+    return IndexedBindingsHandoff(
+        binding_ops=plan,
+        state=state,
+        edge_aliases=tuple(
+            op._name for op in middle
+            if isinstance(op, ASTEdge) and isinstance(op._name, str)
+        ) if state is not None else (),
+    )
+
+
 def _handle_boundary_calls(
     self: Plottable,
     ops: List[ASTObject],
@@ -606,6 +672,10 @@ def _handle_boundary_calls(
     logger.debug('Boundary call pattern detected: prefix=%s, middle=%s, suffix=%s',
                 len(prefix), len(middle), len(suffix))
 
+    # Function-scope import: `gfql.index` transitively imports this module, so a
+    # module-scope import would be a cycle.
+    from .gfql.index.handoff import attach_handoff
+
     g_temp = self
     suffix_base_graph = g_temp
 
@@ -622,24 +692,40 @@ def _handle_boundary_calls(
         )
         suffix_base_graph = g_temp
 
-    if middle:
-        logger.debug('Executing middle operations: %s', middle)
-        g_temp = _chain_impl(
-            g_temp,
-            middle,
-            engine,
-            validate_schema,
-            policy,
-            context,
-            start_nodes
-        )
+    # ONE value carries the whole decision: None = the indexed path does not apply
+    # to this boundary, a handoff WITH state = it serves, a handoff WITHOUT state =
+    # it was tried and declined (which the row materializer must know, so it does
+    # not re-attempt the same plan after the canonical traversal).
+    handoff = _plan_indexed_middle(
+        self, prefix, middle, suffix, engine, policy, start_nodes,
+    )
+    served = handoff is not None and handoff.state is not None
+
+    if served:
+        assert handoff is not None  # narrowed by `served`
+        g_temp = attach_handoff(self, handoff)
+    else:
+        if middle:
+            logger.debug('Executing middle operations: %s', middle)
+            g_temp = _chain_impl(
+                g_temp,
+                middle,
+                engine,
+                validate_schema,
+                policy,
+                context,
+                start_nodes
+            )
+        if handoff is not None:
+            # attach (not mutate): `g_temp` may still be the caller's graph on
+            # paths where nothing rebuilt it, and this is internal plumbing.
+            g_temp = attach_handoff(g_temp, handoff)
 
     if suffix:
         logger.debug('Executing boundary suffix calls: %s', suffix)
         if start_nodes is not None:
-            setattr(g_temp, "_gfql_start_nodes", start_nodes)
-        setattr(g_temp, "_gfql_rows_base_graph", suffix_base_graph)
-        setattr(g_temp, "_gfql_shortest_path_backend", getattr(g_temp, "_gfql_shortest_path_backend", "auto"))
+            g_temp._gfql_start_nodes = start_nodes
+        g_temp._gfql_rows_base_graph = suffix_base_graph
         if (
             middle
             and any(getattr(op, "_name", None) is not None for op in middle)
@@ -898,6 +984,14 @@ def chain(
         # undirected multi-edge); that honest signal propagates to the caller.
         _tgt = ExecutionTarget.GPU if engine_concrete_early == Engine.POLARS_GPU else ExecutionTarget.CPU
         with target_mode(_tgt):
+            if policy:
+                from graphistry.compute.gfql.call.executor import _thread_local as call_thread_local
+                old_policy = getattr(call_thread_local, 'policy', None)
+                try:
+                    call_thread_local.policy = policy
+                    return chain_polars(self, ops, start_nodes=start_nodes)
+                finally:
+                    call_thread_local.policy = old_policy
             return chain_polars(self, ops, start_nodes=start_nodes)
 
     if policy:

graphistry/compute/chain_fast_paths.py

@@ -54,9 +54,11 @@ def _resident_seed_indexes(
     identity via get_valid), else None — callers keep the scan path, so a stale
     or absent index can never change results, only speed."""
     from graphistry.Engine import Engine, is_polars_df
-    from graphistry.compute.gfql.index import get_registry
+    from graphistry.compute.gfql.index import get_index_policy, get_registry
     from graphistry.compute.gfql.index.registry import EDGE_OUT_ADJ, EDGE_IN_ADJ, NODE_ID
     from graphistry.compute.gfql.index.engine_arrays import array_namespace
+    if get_index_policy(g) == "off":
+        return None
     registry = get_registry(g)
     if registry.is_empty():
         return None

graphistry/compute/gfql/cypher/lowering.py

@@ -2130,8 +2130,14 @@ def __init__(self, table_df: Any) -> None:
         self._destination = None
         self._edge = None
         self._g = self
+        # Full GFQL execution context (see Plottable): this stand-in flows through
+        # the same row pipeline, so every field must exist or typed access breaks.
         self._gfql_start_nodes = None
         self._gfql_rows_base_graph = None
+        self._gfql_rows_edge_aliases = None
+        self._gfql_indexed_bindings_handoff = None
+        self._gfql_index_policy = "use"
+        self._gfql_index_registry = None
         self._gfql_shortest_path_backend = "auto"
 
     def bind(self) -> "_SyntheticRowGraph":