perf(gfql): native polars translation for simple connected-join residuals (#1729/#1755)#1763
Merged
Merged
Conversation
lmeyerov
force-pushed
the
perf/gfql-polars-residual-native
branch
from
July 21, 2026 20:57
abc21e0 to
538a9db
Compare
Contributor
Author
|
Review responses:
|
lmeyerov
added a commit
that referenced
this pull request
Jul 21, 2026
…llect (#1755) Profile evidence (dgx, q5 warm): ~27 eager polars ops per execution at a fixed collect cost each = 72% of runtime; all six filter_by_dict calls together were only 1.4ms of 9.5ms, so filter dedup could not win. Instead, when every residual translates natively (the #1763 fast lane), the whole two-star plan — base filters, residual filters, typed-edge filters, semi-joins, group props lookup — now composes lazily and collects ONCE at the join. Value-identical: same filters/joins/aggregation; drops only the leaf-singleton edge prefilter and the all-left-counts==1 early count, both provably result-equivalent to the generic join+sum. filter_by_dict_polars gains a filter_expr_by_dict_polars twin (same resolution + typed-error/NIE contract, expr only). Differential tests: fused vs forced-eager exact-row parity (ORDER BY pinned), empty-result shape, pandas oracle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
force-pushed
the
perf/gfql-polars-residual-native
branch
from
July 21, 2026 21:56
70aadf2 to
10e2685
Compare
…uals (#1729/#1755) The connected-join grouped-count fast path applies scalar residuals (toLower equality, scalar equality/range — the shapes #1729's lowering cannot push into filter_dict) by spinning a where_rows chain per alias (~1.7ms each, the dominant cost of the residual OLAP path: graph-bench q5/q6/q7 spent ~5ms of ~10ms there). On polars, translate exactly those simple shapes directly to native polars filter expressions; ANY other expression falls back to the existing where_rows chain evaluator, so semantics never diverge. Byte-parity: differential fast-vs-fallback identical on 8 adversarial cases (nulls, case variants, unicode casefold, numeric ranges, multi-expr); the full cypher lowering + row-pipeline suites pass (1645, incl. the residual-specific connected-join tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…og (#1729/#1755) Review response: return type was Optional[Any], now Optional['pl.Expr'] via TYPE_CHECKING import; docstring documents why expr is a string (the ASTCall params). Adds the missing CHANGELOG entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…n suite (#1729/#1755) Positive (all covered shapes incl. nulls/casefold/negative literals), negative (unsupported shapes/alias mismatch/absent column decline), the all-or-nothing group fallback gate, and the pandas-frames-never-fast-lane contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
) Review-skill wave findings, empirically confirmed then fixed: - ESCAPED string literals (renderer emits \uXXXX escapes the evaluator unescapes) now DECLINE -> chain fallback compares unescaped correctly (was: silent empty results on e.g. toLower('It\'s')) - dtype-incompatible column/literal pairs now DECLINE so the evaluator raises its designed parity-or-error NotImplementedError (was: raw polars ComputeError); translator now takes the frame schema - NaN-ranking caveat + str.lower()-vs-to_lowercase() documented (NaN unreachable through gfql(): ingest normalizes NaN->null) - merged duplicate is_polars blocks - 6 new gate tests (escape/dtype/categorical/designed-error parity) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…1729/#1755) The changed-line-coverage gate combines core/gfql/polars lane coverage; the polars-only translator lines are exercised only where polars is installed, so the new test file must be in POLARS_TEST_FILES (79.63% -> passes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
force-pushed
the
perf/gfql-polars-residual-native
branch
from
July 21, 2026 22:16
10e2685 to
0662ec2
Compare
lmeyerov
added a commit
that referenced
this pull request
Jul 21, 2026
…llect (#1755) Profile evidence (dgx, q5 warm): ~27 eager polars ops per execution at a fixed collect cost each = 72% of runtime; all six filter_by_dict calls together were only 1.4ms of 9.5ms, so filter dedup could not win. Instead, when every residual translates natively (the #1763 fast lane), the whole two-star plan — base filters, residual filters, typed-edge filters, semi-joins, group props lookup — now composes lazily and collects ONCE at the join. Value-identical: same filters/joins/aggregation; drops only the leaf-singleton edge prefilter and the all-left-counts==1 early count, both provably result-equivalent to the generic join+sum. filter_by_dict_polars gains a filter_expr_by_dict_polars twin (same resolution + typed-error/NIE contract, expr only). Differential tests: fused vs forced-eager exact-row parity (ORDER BY pinned), empty-result shape, pandas oracle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The connected-join grouped-count fast path applies scalar residuals — the shapes
#1729's lowering cannot push into
filter_dict(toLower(a.col) = toLower('lit'),a.col <op> literal) — by spinning awhere_rowschain per alias (~1.7 ms each).On graph-bench q5/q6/q7 that residual-apply was ~5 ms of a ~10 ms query.
On polars, translate exactly those simple shapes directly to native polars filter
expressions (
_residual_polars_expr). Any expression outside the recognizedshapes falls back to the existing
where_rowschain evaluator, so semantics cannever diverge.
Measured (dgx, 20k graph-bench, polars, warm median)
Values unchanged (q5=1111 etc.).
Byte-parity verification
variants, unicode casefold (
Straße), numeric ranges, multi-expr conjunction.residual-specific connected-join tests that caught an earlier (rejected)
implementation approach.
🤖 Generated with Claude Code