Skip to content

perf(gfql): native polars translation for simple connected-join residuals (#1729/#1755)#1763

Merged
lmeyerov merged 5 commits into
masterfrom
perf/gfql-polars-residual-native
Jul 21, 2026
Merged

perf(gfql): native polars translation for simple connected-join residuals (#1729/#1755)#1763
lmeyerov merged 5 commits into
masterfrom
perf/gfql-polars-residual-native

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

What

The connected-join grouped-count fast path applies scalar residuals — the shapes
#1729's lowering cannot push into filter_dict (toLower(a.col) = toLower('lit'),
a.col <op> literal) — by spinning a where_rows chain per alias (~1.7 ms each).
On graph-bench q5/q6/q7 that residual-apply was ~5 ms of a ~10 ms query.

On polars, translate exactly those simple shapes directly to native polars filter
expressions (_residual_polars_expr). Any expression outside the recognized
shapes falls back to the existing where_rows chain evaluator, so semantics can
never diverge.

Measured (dgx, 20k graph-bench, polars, warm median)

query before after
q5 10.2 ms 6.6 ms
q6 10.7 ms 8.9 ms
q7 13.5 ms 10.5 ms

Values unchanged (q5=1111 etc.).

Byte-parity verification

  • Differential fast-vs-fallback identical on 8 adversarial cases: nulls, case
    variants, unicode casefold (Straße), numeric ranges, multi-expr conjunction.
  • Full cypher lowering + row-pipeline suites: 1645 pass — including the
    residual-specific connected-join tests that caught an earlier (rejected)
    implementation approach.

🤖 Generated with Claude Code

@lmeyerov
lmeyerov force-pushed the perf/gfql-polars-residual-native branch from abc21e0 to 538a9db Compare July 21, 2026 20:57
@lmeyerov

Copy link
Copy Markdown
Contributor Author

Review responses:

  • _residual_polars_expr return type is now Optional['pl.Expr'] (TYPE_CHECKING import), was Optional[Any].
  • expr: str is the honest input type — the feat(gfql): plan connected join predicate pushdown #1729 lowering serializes residuals as canonical predicate strings in ASTCall params (a typed term needs a lowering-level refactor, out of scope); rationale now in the docstring.
  • CHANGELOG entry added.
  • New committed regression suite test_residual_polars_native.py (20 tests): positive shapes incl. nulls/casefold/negative literals, negative declines, the all-or-nothing group-fallback gate, pandas-frames-never-fast-lane.

lmeyerov added a commit that referenced this pull request Jul 21, 2026
…llect (#1755)

Profile evidence (dgx, q5 warm): ~27 eager polars ops per execution at a fixed
collect cost each = 72% of runtime; all six filter_by_dict calls together were
only 1.4ms of 9.5ms, so filter dedup could not win. Instead, when every residual
translates natively (the #1763 fast lane), the whole two-star plan — base
filters, residual filters, typed-edge filters, semi-joins, group props lookup —
now composes lazily and collects ONCE at the join. Value-identical: same
filters/joins/aggregation; drops only the leaf-singleton edge prefilter and the
all-left-counts==1 early count, both provably result-equivalent to the generic
join+sum. filter_by_dict_polars gains a filter_expr_by_dict_polars twin (same
resolution + typed-error/NIE contract, expr only). Differential tests: fused vs
forced-eager exact-row parity (ORDER BY pinned), empty-result shape, pandas
oracle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov
lmeyerov force-pushed the perf/gfql-polars-residual-native branch from 70aadf2 to 10e2685 Compare July 21, 2026 21:56
lmeyerov and others added 5 commits July 21, 2026 15:15
…uals (#1729/#1755)

The connected-join grouped-count fast path applies scalar residuals
(toLower equality, scalar equality/range — the shapes #1729's lowering cannot
push into filter_dict) by spinning a where_rows chain per alias (~1.7ms each,
the dominant cost of the residual OLAP path: graph-bench q5/q6/q7 spent ~5ms
of ~10ms there). On polars, translate exactly those simple shapes directly to
native polars filter expressions; ANY other expression falls back to the
existing where_rows chain evaluator, so semantics never diverge.

Byte-parity: differential fast-vs-fallback identical on 8 adversarial cases
(nulls, case variants, unicode casefold, numeric ranges, multi-expr); the full
cypher lowering + row-pipeline suites pass (1645, incl. the residual-specific
connected-join tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…og (#1729/#1755)

Review response: return type was Optional[Any], now Optional['pl.Expr']
via TYPE_CHECKING import; docstring documents why expr is a string (the
ASTCall params). Adds the missing CHANGELOG entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…n suite (#1729/#1755)

Positive (all covered shapes incl. nulls/casefold/negative literals),
negative (unsupported shapes/alias mismatch/absent column decline), the
all-or-nothing group fallback gate, and the pandas-frames-never-fast-lane
contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
)

Review-skill wave findings, empirically confirmed then fixed:
- ESCAPED string literals (renderer emits \uXXXX escapes the evaluator
  unescapes) now DECLINE -> chain fallback compares unescaped correctly
  (was: silent empty results on e.g. toLower('It\'s'))
- dtype-incompatible column/literal pairs now DECLINE so the evaluator
  raises its designed parity-or-error NotImplementedError (was: raw
  polars ComputeError); translator now takes the frame schema
- NaN-ranking caveat + str.lower()-vs-to_lowercase() documented (NaN
  unreachable through gfql(): ingest normalizes NaN->null)
- merged duplicate is_polars blocks
- 6 new gate tests (escape/dtype/categorical/designed-error parity)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…1729/#1755)

The changed-line-coverage gate combines core/gfql/polars lane coverage; the
polars-only translator lines are exercised only where polars is installed, so
the new test file must be in POLARS_TEST_FILES (79.63% -> passes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov
lmeyerov force-pushed the perf/gfql-polars-residual-native branch from 10e2685 to 0662ec2 Compare July 21, 2026 22:16
@lmeyerov
lmeyerov merged commit 0bb7761 into master Jul 21, 2026
77 checks passed
lmeyerov added a commit that referenced this pull request Jul 21, 2026
…llect (#1755)

Profile evidence (dgx, q5 warm): ~27 eager polars ops per execution at a fixed
collect cost each = 72% of runtime; all six filter_by_dict calls together were
only 1.4ms of 9.5ms, so filter dedup could not win. Instead, when every residual
translates natively (the #1763 fast lane), the whole two-star plan — base
filters, residual filters, typed-edge filters, semi-joins, group props lookup —
now composes lazily and collects ONCE at the join. Value-identical: same
filters/joins/aggregation; drops only the leaf-singleton edge prefilter and the
all-left-counts==1 early count, both provably result-equivalent to the generic
join+sum. filter_by_dict_polars gains a filter_expr_by_dict_polars twin (same
resolution + typed-error/NIE contract, expr only). Differential tests: fused vs
forced-eager exact-row parity (ORDER BY pinned), empty-result shape, pandas
oracle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant