perf(gfql): route engine=auto to native polars for polars-frame graphs#1743
Draft
lmeyerov wants to merge 3 commits into
Draft
perf(gfql): route engine=auto to native polars for polars-frame graphs#1743lmeyerov wants to merge 3 commits into
lmeyerov wants to merge 3 commits into
Conversation
resolve_engine(AUTO) maps polars frames to PANDAS (it predates Engine.POLARS), so g.gfql(query) on a polars-frame graph silently bridged to the generic pandas path: ~3-13x slower on cypher point queries and pandas frames out. Route AUTO to the native polars engine; an honest NotImplementedError (unsupported shape) falls back to the legacy AUTO path — allowed because the user did not pin an engine. Frames in = frames out: AUTO results on polars graphs are now polars. Repro: 2k-node polars graph, seeded 1-hop cypher — AUTO 25.0ms/pandas out before, 9.3ms/polars out after (engine='polars' = 8.8ms); polars-NIE shapes (shortestPath) still answer via the pandas fallback. Fixes the q5 finding in plans/gfql-benchmark-numbers (inferred-engine 13x penalty). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…c records The dual-path guards checked hasattr(out, 'to_dict') first, but polars DataFrame HAS to_dict (without orient) — so the polars branch was unreachable and these tests only ever ran because auto silently returned pandas. With auto routing polars-frame graphs natively they now receive polars frames; probe to_dicts first. Assertions (record values) unchanged and still pass on both engines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
marked this pull request as draft
July 21, 2026 08:51
Contributor
Author
|
Too early to enable by default |
lmeyerov
added a commit
that referenced
this pull request
Jul 22, 2026
…rialize-first Review findings (plans/review-pr-1767/review.md): B1 edges-only polars graph crashed in the NODE_ID build (materialize under AUTO synthesizes pandas; polars gathers ran on a pandas frame) — gfql_index_all now materializes + engine-aligns BEFORE any build so all indexes land valid, and create_index re-aligns for direct calls. M1 LazyFrame frames under AUTO crashed — the gate now requires EAGER polars DataFrames on ALL present frames (LazyFrame/mixed keep the legacy pandas path). Spy test asserts the index SERVED (non-None), not merely was called. AUTO-query trade-off documented in CHANGELOG (deferred to #1743 by policy hold). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
resolve_engine(AUTO)maps polars frames toEngine.PANDAS(that branch predatesEngine.POLARS), sog.gfql(query)on a polars-frame graph silently bridges to the generic pandas path — measured ~3× on a small seeded cypher query and ~13× on the q5 graph-benchmark shape (seeplans/gfql-benchmark-numbers), with pandas frames coming back out.This routes
engine='auto'(the default) to the native polars engine when both bound frames are polars, with an honest-NotImplementedErrorfallback to the legacy AUTO path — the fallback is permitted here because the user did not pin an engine (the no-silent-fallback contract applies to explicitengine=).gfql()): changingresolve_engineitself would ripple into plotting/util paths that only handle pandas/cudf.Measurements (2k-node polars graph, seeded 1-hop cypher, warm)
'polars'(explicit)Tests
TestAutoEngineRoutesPolarsNative: auto == explicit-polars result (frame-equal, native frames); shortestPath NIEs when pinned but answers via auto fallback.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL