Search before asking
Description
Today ORC/Parquet dictionary filtering only applies when a filter conjunct is a bare column reference compared to constants. A predicate whose value side is a deterministic single-slot string expression — e.g. split_by_string(col, sep)[n] = 'x', element_at(...), substring_index(...), regexp_extract(...) — is evaluated once per row even though it depends only on the column value.
Such predicates can instead be evaluated once per distinct dictionary value. The FE (Nereids) is the authoritative admission check: it stamps a conjunct root as dict-filterable when the predicate is an equality/IN over a single slot, is deterministic, and preserves NULL semantics (null-in implies null-out — the dictionary carries no NULL entry, so a NULL source must map to a NULL result to match per-row semantics). The verdict is carried to BE via TExprNode.can_dict_filter; when unset (older FE) BE falls back to a conservative allowlist. A session variable enable_dict_filter_for_expr (default true, forwarded as a query option) gates the expression case.
Use case
Scans that filter on a heavy string expression of a low-cardinality dictionary-encoded ORC/Parquet column evaluate the expression per row today; evaluating per distinct dictionary value is a large reduction for low-NDV columns.
Related issues
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
Today ORC/Parquet dictionary filtering only applies when a filter conjunct is a bare column reference compared to constants. A predicate whose value side is a deterministic single-slot string expression — e.g.
split_by_string(col, sep)[n] = 'x',element_at(...),substring_index(...),regexp_extract(...)— is evaluated once per row even though it depends only on the column value.Such predicates can instead be evaluated once per distinct dictionary value. The FE (Nereids) is the authoritative admission check: it stamps a conjunct root as dict-filterable when the predicate is an equality/IN over a single slot, is deterministic, and preserves NULL semantics (null-in implies null-out — the dictionary carries no NULL entry, so a NULL source must map to a NULL result to match per-row semantics). The verdict is carried to BE via
TExprNode.can_dict_filter; when unset (older FE) BE falls back to a conservative allowlist. A session variableenable_dict_filter_for_expr(default true, forwarded as a query option) gates the expression case.Use case
Scans that filter on a heavy string expression of a low-cardinality dictionary-encoded ORC/Parquet column evaluate the expression per row today; evaluating per distinct dictionary value is a large reduction for low-NDV columns.
Related issues
No response
Are you willing to submit PR?
Code of Conduct