[AURON #2425] Support Iceberg STARTS_WITH pruning in native scan#2426
Conversation
Signed-off-by: weimingdiit <weimingdiit@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds support for translating Iceberg STARTS_WITH residual predicates into Spark StartsWith expressions so they can be converted into native scan pruning predicates, improving pruning for prefix LIKE 'x%' queries in Iceberg native scans.
Changes:
- Convert Iceberg
STARTS_WITHonStringTypeinto SparkStartsWithfor scan-pruning conversion. - Add an Iceberg integration test validating that prefix
LIKEproduces a native scan pruning predicate usingstarts_with.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala | Adds Iceberg STARTS_WITH → Spark StartsWith mapping for scan pruning predicate generation. |
| thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala | Adds an integration test asserting LIKE 'a%' results in a starts_with pruning predicate in the native scan plan. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| val pruningPredicateText = nativeScanPlan.get.pruningPredicates.mkString("\n") | ||
| assert( | ||
| pruningPredicateText.contains("name: \"starts_with\"") && | ||
| pruningPredicateText.contains("fun: StartsWith"), | ||
| pruningPredicateText) |
slfan1989
left a comment
There was a problem hiding this comment.
Thanks for adding Iceberg STARTS_WITH pruning support. The implementation is focused and reuses the existing native conversion path well. It would be better to assert the structured protobuf fields in the test instead of matching its toString output, but this is non-blocking.
|
@weimingdiit Thanks for the contrubution! Merged into master. |
Which issue does this PR close?
Closes #2425
Rationale for this change
Auron Iceberg native scan already collects Iceberg residual filters and converts supported filters into native scan pruning predicates.
Iceberg can produce
STARTS_WITHfilters for prefix string predicates, such asLIKE 'a%', but Auron currently does not convert this Iceberg operation. As a result, the query can still use a native post-scan filter, but the starts-with predicate is not used for native scan pruning.What changes are included in this PR?
Adds Iceberg
STARTS_WITHconversion for string columns inIcebergScanSupport.Reuses the existing Spark
StartsWithnative pruning conversion.Keeps other string and binary pruning behavior unchanged.
Adds an Iceberg integration test for prefix LIKE pruning.
Are there any user-facing changes?
No user-facing API changes. More Iceberg prefix string filters can now be passed into native scan pruning.
How was this patch tested?
UT.