perf(gfql): fuse residual two-star grouped-count into one lazy collect — q5/q6/q7 flip to wins (#1755)#1764
Merged
Merged
Conversation
lmeyerov
force-pushed
the
perf/gfql-polars-residual-native
branch
2 times, most recently
from
July 21, 2026 22:16
10e2685 to
0662ec2
Compare
lmeyerov
changed the base branch from
perf/gfql-polars-residual-native
to
master
July 21, 2026 22:33
…llect (#1755) Profile evidence (dgx, q5 warm): ~27 eager polars ops per execution at a fixed collect cost each = 72% of runtime; all six filter_by_dict calls together were only 1.4ms of 9.5ms, so filter dedup could not win. Instead, when every residual translates natively (the #1763 fast lane), the whole two-star plan — base filters, residual filters, typed-edge filters, semi-joins, group props lookup — now composes lazily and collects ONCE at the join. Value-identical: same filters/joins/aggregation; drops only the leaf-singleton edge prefilter and the all-left-counts==1 early count, both provably result-equivalent to the generic join+sum. filter_by_dict_polars gains a filter_expr_by_dict_polars twin (same resolution + typed-error/NIE contract, expr only). Differential tests: fused vs forced-eager exact-row parity (ORDER BY pinned), empty-result shape, pandas oracle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
force-pushed
the
perf/gfql-olap-q5-filter-dedup
branch
from
July 21, 2026 22:33
749ab72 to
4abe2a1
Compare
…y, engagement-asserted tests
Review-skill blockers (plans/review-pr-1764/review.md):
B1 fused lane called .lazy() on unconverted edges -> AttributeError on
pandas edges + engine='polars' (incl. WITH..MATCH reentry). Edges now
df_to_engine'd for the lane.
B2 dropped all-left-counts==1 shortcut was not result-equivalent on the
empty-join boundary: ungrouped RETURN count(p) with a live first arm
returned [] instead of the eager lane's [{'n':0}] (openCypher count over
no rows). left_counts rides the same collect_all; boundary reproduced.
B3 TestFusedTwoStarLane was vacuous (count(*) declines the whole two-star
path at the agg gate). Lane extracted to a module-level helper
(_connected_join_two_star_fused_polars); every fused-arm test now spies
it and asserts engagement; count(*) pinned as a decline shape; new B1/B2
repro pins. Also: LazyFrame inputs decline to the eager lane; pl.Expr/
pl.LazyFrame typings; honest CHANGELOG wording.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…oundary collect_all of both plans cost +2.5ms/query on 20k graphbench q5-q7 (CSE does not absorb the left-arm recompute); the hot path is back to one collect, the boundary pays a second collect only when the join is empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1763 (native residual translation) — review/merge that first.
Problem
Profiling q5 on dgx showed the residual two-star grouped-count path spends ~72% of runtime on the FIXED collect cost of ~27 eager polars ops per execution (270 collects over 10 execs = 66ms of 92ms). The filters themselves — the previously-planned dedup target — totaled only 1.4ms of 9.5ms, so filter caching could not win (measured: it regressed).
Fix
When every residual translates natively (the #1763 fast lane), the whole two-star plan — base
filter_dicts, residual filters, typed-edge filters, semi-joins, group-prop lookup — composes as ONEpl.LazyFrameplan, collected once at the join. The tail (group_by/sort/head/select on the tiny joined frame) is unchanged.filter_by_dict_polarsis factored into itself +filter_expr_by_dict_polars(same column/dtype resolution, same typed-error/NIE contract, expression only) so the lazy plan reuses the exact filter semantics.Value-identical: same filters/joins/aggregation. Two perf-only shortcuts are dropped in the fused lane, both provably result-equivalent to the generic join+sum path: the leaf-singleton edge prefilter (same surviving rows) and the all-left-counts==1 early count (sum of ones == count).
Measured (dgx, same-session, warm, 15 runs; values byte-identical)
Graph-benchmark scoreboard moves from 4W/2T/3L to 7W/1T/1L (remaining loss: q4, 6.1 vs 5.1; q3 ~tie).
Testing
TestFusedTwoStarLane: fused vs forced-eager exact-row parity (ORDER BY pinned), empty-result shape parity, pandas oracle.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL