Skip to content

[Sandbox SQL snapshot] Pin org.opensearch.query:* (unified-query-*) snapshots to mavenLocal#21578

Merged
mch2 merged 2 commits intoopensearch-project:mainfrom
RyanL1997:fix-mavenlocal-pin-unified-query
May 9, 2026
Merged

[Sandbox SQL snapshot] Pin org.opensearch.query:* (unified-query-*) snapshots to mavenLocal#21578
mch2 merged 2 commits intoopensearch-project:mainfrom
RyanL1997:fix-mavenlocal-pin-unified-query

Conversation

@RyanL1997
Copy link
Copy Markdown
Contributor

@RyanL1997 RyanL1997 commented May 9, 2026

Description

Follow-up to #21569. The sandbox-check workflow's pre-step publishUnifiedQueryPublicationToMavenLocal publishes the SQL feature branch's unified-query-* jars to mavenLocal so analytics-engine sandbox plugins build against the latest API surface. The publish itself works, but Gradle's default SNAPSHOT resolution still prefers the stale remote — every test that exercises the unified-query path picks up jars built from sql/main, not the feature branch.

Confirmed via dependencyInsight

$ ./gradlew :sandbox:plugins:test-ppl-frontend:dependencyInsight \
    --dependency unified-query-api --configuration runtimeClasspath \
    --refresh-dependencies
org.opensearch.query:unified-query-api:3.7.0.0-SNAPSHOT:20260507.224009-12   ← REMOTE timestamp

Why mavenLocal loses

Comparing maven-metadata.xml from each source for unified-query-api:3.7.0.0-SNAPSHOT:

Source <lastUpdated> <buildNumber> Identifier
mavenLocal (post-publish) 20260508200435 (newer) none, just <localCopy>true</localCopy> bare 3.7.0.0-SNAPSHOT
Remote OpenSearch Snapshots 20260507224009 (older) 12 timestamped 3.7.0.0-20260507.224009-12

Gradle's resolver weighs the remote's explicit <buildNumber> + timestamp metadata higher than mavenLocal's bare <localCopy>true>, so the older remote wins even when mavenLocal is newer. End-to-end consequence: test-ppl-frontend bundles the stale 60kB unified-query-api-3.7.0.0-SNAPSHOT.jar (42 classes) instead of the freshly-published 29kB feature-branch jar (21 classes), and the runtime cluster ends up running the older UnifiedQueryContext$Builder that's missing PPL_REX_MAX_MATCH_LIMIT, CALCITE_ENGINE_ENABLED, and any other field the feature branch has added but sql/main hasn't received yet.

This causes RexCommandIT and any other IT exercising rex max-match (or anything else introduced post-sql/main) to fail with Cannot invoke 'java.lang.Integer.intValue()' because Settings.getSettingValue(...) is null at plan time, even though the workflow publish step succeeds.

Fix

Tell Gradle's repository content filters that org.opensearch.query is owned by mavenLocal:

  • sandbox/build.gradle subprojects { repositories } — add mavenLocal { mavenContent { includeGroup 'org.opensearch.query' } } and add mavenContent { excludeGroup 'org.opensearch.query' } to the existing OpenSearch Snapshots remote. Every sandbox subproject now resolves the unified-query group exclusively from mavenLocal; everything else still goes to the remote unchanged.
  • sandbox/plugins/analytics-backend-datafusion/build.gradle — same excludeGroup filter on its own remote declaration (the file-local block isn't covered by the parent subprojects block, which only adds repos rather than replacing them).
  • sandbox/plugins/test-ppl-frontend/build.gradle — same includeGroup / excludeGroup pair locally so bundlePlugin actually picks the mavenLocal jar at bundle time.
  • sandbox/plugins/analytics-engine/build.gradle — bump sqlUnifiedQueryVersion from 3.6.0.0-SNAPSHOT3.7.0.0-SNAPSHOT to match test-ppl-frontend. The 3.6 line was stale; with the new filter the remote no longer serves it, and the older pin caused Could not find org.opensearch.query:unified-query-api:3.6.0.0-SNAPSHOT at internal-cluster-test resolution.

Verification

  • ./gradlew :sandbox:plugins:test-ppl-frontend:bundlePlugin --refresh-dependencies — bundled unified-query-api-3.7.0.0-SNAPSHOT.jar drops 60kB → 29kB and the constant pool now references PPL_REX_MAX_MATCH_LIMIT.
  • ./gradlew :sandbox:qa:analytics-engine-rest:integTest --tests RexCommandIT — 0/16 → 16/16.
  • ./gradlew :sandbox:plugins:analytics-engine:compileJava :sandbox:plugins:analytics-engine:compileTestJava — green (the 3.7 bump resolves cleanly through mavenLocal now).

Drop conditions

This is a temporary workaround mirroring the temporary nature of #21569. Drop both filters once the SQL feature branch merges to sql/main and the OpenSearch Snapshots remote starts republishing the unified-query group. At that point Gradle resolution will be correct without any content-filter intervention.

RyanL1997 added 2 commits May 9, 2026 00:33
The remote OpenSearch Snapshots maven repo (ci.opensearch.org/ci/dbc/snapshots)
only republishes from sql/main, not from sql/feature/mustang-ppl-integration,
so its 3.7.0.0-SNAPSHOT jars trail the feature branch by however many merges
(currently missing PPL_REX_MAX_MATCH_LIMIT, CALCITE_ENGINE_ENABLED, …). The
sandbox-check workflow's pre-step opensearch-project#21569 publishes feature-branch unified-query
jars to mavenLocal, but Gradle's default SNAPSHOT resolution weighs the remote's
explicit <buildNumber>/<timestamp> metadata higher than mavenLocal's
<localCopy>true>, so the stale remote wins even when mavenLocal has a newer
<lastUpdated>.

Confirmed via dependencyInsight: every consumer was binding
unified-query-api:3.7.0.0-SNAPSHOT:20260507.224009-12 (60kB, 42 classes, no
PPL_REX_MAX_MATCH_LIMIT field reference) instead of the locally-published
3.7.0.0-SNAPSHOT (29kB, 21 classes, has the field). The runtime cluster
inherited that stale class via the test-ppl-frontend plugin bundle, which
is why every IT touching `rex` failed plan-time with `NullPointerException:
Cannot invoke "java.lang.Integer.intValue()" because the return value of
"Settings.getSettingValue(PPL_REX_MAX_MATCH_LIMIT)" is null` once the
unified path tried to read the setting.

Fix: tell the OpenSearch Snapshots remote to refuse `org.opensearch.query`
artifacts via mavenContent { excludeGroup }. Three sites declare the remote:

  * sandbox/build.gradle subprojects { repositories } — applies to every
    sandbox subproject including qa.
  * sandbox/plugins/analytics-backend-datafusion/build.gradle — own
    declaration; left in place for module isolation, filtered identically.
  * sandbox/plugins/test-ppl-frontend/build.gradle — also pin mavenLocal as
    the only source for org.opensearch.query so the bundlePlugin task
    bundles the freshly-published feature-branch jar rather than the stale
    timestamped one Gradle would otherwise pick.

Verified locally: bundled unified-query-api drops 60kB → 29kB, the
UnifiedQueryContext$Builder constant pool now references PPL_REX_MAX_MATCH_LIMIT,
and RexCommandIT goes 0/16 → 16/16 against the same locally-published jars
the CI workflow already produces.

Drop this filter once the SQL feature branch merges to sql/main and the
remote OpenSearch Snapshots repo catches up — at that point every
3.7.0.0-SNAPSHOT publish will carry the rex max-match default and the
mavenLocal preference becomes redundant.

Signed-off-by: Jialiang Liang <jiallian@amazon.com>
CI fallout from the prior commit's `excludeGroup 'org.opensearch.query'`
filter on the OpenSearch Snapshots remote: the parent subprojects block
no longer carried mavenLocal, so analytics-engine's testImplementation /
internalClusterTest configurations had no repository at all serving
org.opensearch.query, failing with `Could not find
org.opensearch.query:unified-query-api:3.6.0.0-SNAPSHOT` (and -core / -ppl).

Two pieces:

1. sandbox/build.gradle subprojects { repositories } — also declare
   mavenLocal scoped to the org.opensearch.query group via mavenContent
   { includeGroup }. mavenLocal becomes the authoritative source for
   unified-query SNAPSHOTs (populated by the sandbox-check workflow's
   publishUnifiedQueryPublicationToMavenLocal pre-step) without leaking
   into resolution for any other group.

2. sandbox/plugins/analytics-engine/build.gradle — bump
   sqlUnifiedQueryVersion from 3.6.0.0-SNAPSHOT → 3.7.0.0-SNAPSHOT.
   The 3.6 jars don't exist in mavenLocal (only the 3.7 feature-branch
   build does), so the older pin was the proximate cause of the CI
   resolution failure. Aligning with test-ppl-frontend's already-3.7
   declaration also keeps the unified-query consumers consistent.

Signed-off-by: Jialiang Liang <jiallian@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 84bfec6.

PathLineSeverityDescription
sandbox/build.gradle57highRepository source change: mavenLocal() added with includeGroup filter for 'org.opensearch.query'. mavenLocal resolves artifacts from the local filesystem (~/.m2/repository), a known supply chain attack vector in CI/CD environments where an attacker with filesystem access can inject malicious artifacts that match the expected group/artifact coordinates.
sandbox/build.gradle63highRepository resolution change: the existing remote OpenSearch Snapshots Maven repo is now reconfigured to exclude 'org.opensearch.query', redirecting all resolution of that group to mavenLocal. This changes the authoritative source of dependency artifacts from a remote server to the local filesystem for this entire group.
sandbox/plugins/analytics-backend-datafusion/build.gradle19highRepository configuration change: remote OpenSearch Snapshots Maven repo is modified to exclude 'org.opensearch.query', altering how dependency artifacts for that group are resolved in this subproject. Any dependency resolution config change must be flagged per mandatory rule.
sandbox/plugins/analytics-engine/build.gradle22highDependency version bump: sqlUnifiedQueryVersion changed from '3.6.0.0-SNAPSHOT' to '3.7.0.0-SNAPSHOT'. Combined with the repository routing changes that pin this group to mavenLocal, the artifact at this new version is sourced entirely from the local Maven cache populated by a CI pre-step, not from a remotely auditable registry.
sandbox/plugins/test-ppl-frontend/build.gradle35highRepository configuration change: the previously unrestricted mavenLocal() declaration is replaced with a filtered mavenLocal (includeGroup 'org.opensearch.query') and a new remote Maven repository is added. This restructures artifact resolution for this subproject — SNAPSHOT artifacts for org.opensearch.query now come exclusively from the local filesystem.

The table above displays the top 10 most important findings.

Total: 5 | Critical: 0 | High: 5 | Medium: 0 | Low: 0


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

@RyanL1997
Copy link
Copy Markdown
Contributor Author

Currently running tests on my rex PR: #21550, this should temporally fix the sql snapshot issue for sandbox testing workflow.

cc @mch2

@RyanL1997
Copy link
Copy Markdown
Contributor Author

RyanL1997 commented May 9, 2026

Confirmed the above change fixed the issue - here is a green run example on rex PR: cae2cb0

@RyanL1997 RyanL1997 marked this pull request as ready for review May 9, 2026 07:49
@RyanL1997 RyanL1997 requested a review from a team as a code owner May 9, 2026 07:49
@RyanL1997 RyanL1997 changed the title [Sandbox] Pin org.opensearch.query:* (unified-query-*) snapshots to mavenLocal [Sandbox SQL sanpshot] Pin org.opensearch.query:* (unified-query-*) snapshots to mavenLocal May 9, 2026
@RyanL1997 RyanL1997 changed the title [Sandbox SQL sanpshot] Pin org.opensearch.query:* (unified-query-*) snapshots to mavenLocal [Sandbox SQL snapshot] Pin org.opensearch.query:* (unified-query-*) snapshots to mavenLocal May 9, 2026
@mch2 mch2 added skip-diff-analyzer Maintainer to skip code-diff-analyzer check, after reviewing issues in AI analysis. skip-diff-reviewer Maintainer to skip code-diff-reviewer check, after reviewing issues in AI analysis. labels May 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

✅ Gradle check result for 84bfec6: SUCCESS

@codecov
Copy link
Copy Markdown

codecov Bot commented May 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.52%. Comparing base (8f72a95) to head (84bfec6).

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #21578      +/-   ##
============================================
+ Coverage     73.48%   73.52%   +0.03%     
+ Complexity    74646    74620      -26     
============================================
  Files          5980     5980              
  Lines        338777   338777              
  Branches      48848    48848              
============================================
+ Hits         248964   249097     +133     
+ Misses        70026    69788     -238     
- Partials      19787    19892     +105     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mch2 mch2 merged commit 36809cc into opensearch-project:main May 9, 2026
38 of 40 checks passed
mch2 added a commit that referenced this pull request May 9, 2026
…ery-*) snapshots to mavenLocal (#21578)" (#21580)

This reverts commit 36809cc.

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
bowenlan-amzn added a commit to bowenlan-amzn/OpenSearch that referenced this pull request May 10, 2026
These dependencies (unified-query-api/core/ppl) were added in opensearch-project#21578 and
then reverted in opensearch-project#21580. They were incorrectly re-introduced during
conflict resolution.

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-diff-analyzer Maintainer to skip code-diff-analyzer check, after reviewing issues in AI analysis. skip-diff-reviewer Maintainer to skip code-diff-reviewer check, after reviewing issues in AI analysis.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants