Skip to content

[AURON #2425] Support Iceberg STARTS_WITH pruning in native scan#2426

Merged
slfan1989 merged 1 commit into
apache:masterfrom
weimingdiit:fix/iceberg-starts-with-pruning
Jul 26, 2026
Merged

[AURON #2425] Support Iceberg STARTS_WITH pruning in native scan#2426
slfan1989 merged 1 commit into
apache:masterfrom
weimingdiit:fix/iceberg-starts-with-pruning

Conversation

@weimingdiit

Copy link
Copy Markdown
Contributor

Which issue does this PR close?
Closes #2425

Rationale for this change
Auron Iceberg native scan already collects Iceberg residual filters and converts supported filters into native scan pruning predicates.

Iceberg can produce STARTS_WITH filters for prefix string predicates, such as LIKE 'a%', but Auron currently does not convert this Iceberg operation. As a result, the query can still use a native post-scan filter, but the starts-with predicate is not used for native scan pruning.

What changes are included in this PR?
Adds Iceberg STARTS_WITH conversion for string columns in IcebergScanSupport.
Reuses the existing Spark StartsWith native pruning conversion.
Keeps other string and binary pruning behavior unchanged.
Adds an Iceberg integration test for prefix LIKE pruning.

Are there any user-facing changes?
No user-facing API changes. More Iceberg prefix string filters can now be passed into native scan pruning.

How was this patch tested?
UT.

Signed-off-by: weimingdiit <weimingdiit@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for translating Iceberg STARTS_WITH residual predicates into Spark StartsWith expressions so they can be converted into native scan pruning predicates, improving pruning for prefix LIKE 'x%' queries in Iceberg native scans.

Changes:

  • Convert Iceberg STARTS_WITH on StringType into Spark StartsWith for scan-pruning conversion.
  • Add an Iceberg integration test validating that prefix LIKE produces a native scan pruning predicate using starts_with.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala Adds Iceberg STARTS_WITH → Spark StartsWith mapping for scan pruning predicate generation.
thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala Adds an integration test asserting LIKE 'a%' results in a starts_with pruning predicate in the native scan plan.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +483 to +487
val pruningPredicateText = nativeScanPlan.get.pruningPredicates.mkString("\n")
assert(
pruningPredicateText.contains("name: \"starts_with\"") &&
pruningPredicateText.contains("fun: StartsWith"),
pruningPredicateText)
@slfan1989 slfan1989 self-assigned this Jul 26, 2026

@slfan1989 slfan1989 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding Iceberg STARTS_WITH pruning support. The implementation is focused and reuses the existing native conversion path well. It would be better to assert the structured protobuf fields in the test instead of matching its toString output, but this is non-blocking.

@slfan1989
slfan1989 merged commit fe9a2ff into apache:master Jul 26, 2026
123 checks passed
@slfan1989

Copy link
Copy Markdown
Contributor

@weimingdiit Thanks for the contrubution! Merged into master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Iceberg STARTS_WITH pruning in native scan

3 participants