[GLUTEN-12599][VL] Fall back to actions/cache when Apache Stash misses#12623
[GLUTEN-12599][VL] Fall back to actions/cache when Apache Stash misses#12623Smallfu666 wants to merge 3 commits into
Conversation
Per review: Stash has no restore-keys, so a changed ep/build-velox/src tree misses its exact key and cold-builds. Keep the Stash restore first on the x86/arm/enhanced consumers, and fall back to actions/cache/restore (with restore-keys, warm from main) when stash-hit is false; the actions/cache save is retained so both paths stay populated. Producers, nightly and bundle keep the legacy actions/cache method unchanged. Also add a default branch to the gh/jq arch case per review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Appends -forced-miss-validation to all 6 consumer Stash restore keys so the fallback actions/cache/restore is exercised during CI. Will be reverted before the PR review.
|
@Smallfu666 thanks a lot for the contribution! I have included this commit in (#12615) to speed up a bit on the rebase work |
|
@zhouyuan Thanks a lot for the review and for including the commit in #12615 — glad it was useful. I’m still new to Gluten and currently learning the project through the Velox CI/cache work and the Spark–Velox correctness side. I’d also like to understand the project’s current direction better. Are there particular Velox backend areas or ongoing priorities where more contributor help would be especially useful? |
This reverts commit b1365d1.
|
@Smallfu666 If you're looking for a specific place to dive in, the ANSI support issue (#10134) is a good choice. You can start by reading the Spark code for those functions and then implementing them in the Velox backend. |
What changes are proposed in this pull request?
Follow-up to #12610 (merged as 86df856). Apache Stash only supports exact-key restore and has no
restore-keysmechanism. WhenhashFiles('ep/build-velox/src/**')changes for a PR branch, the exact Stash key misses and the native build starts cold. This PR adds a fallback to the legacyactions/cache/restorewith the originalrestore-keysprefix, which can still pull the latest warm cache seeded onmain.Design
Each consumer restore now works in two steps:
stash/restorefirst (exactccache-<variant>-${{ hashFiles('ep/build-velox/src/**') }}key);stash-hitis not'true', fall back toactions/cache/restorewith the pre-migrationrestore-keys, which pulls the latest cache seeded on main.The existing
actions/cache/savesteps are kept, so both the Stash and actions/cache paths stay populated.Scope
velox_backend_arm.yml,velox_backend_enhanced.yml,velox_backend_x86.ymlvelox_backend_cache.yml(producer),velox_nightly.yml,build_bundle_package.yml,flink.yml,velox_backend_ansi.ymlThe cache producer, nightly and bundle workflows, and self-contained caches (flink, ansi) are untouched — they keep the legacy
actions/cachemethod as-is.How was this patch tested?
Static validation
restore-keysprefixes have been verified against the producer'sactions/cache/saveentries invelox_backend_cache.yml.Forced-miss CI validation: PASS
A temporary commit appended
-forced-miss-validationto the six Stash restore keys to force Stash to miss. All three Velox Backend workflows completed successfully:stash-hit=falseccache-centos7-release-default)stash-hit=falseccache-centos8-release-default-ARM64, 443 MB, ~3.2 s)stash-hit=falseccache-enhanced-centos7-release-default)Key evidence from each workflow:
Get Ccache from Apache Stashstep loggedstash-hit=false— the key suffix-forced-miss-validationwas present.Get Ccache from actions/cache (fallback on Stash miss)step executed and restored a warm cache via therestore-keysprefix."Successfully built Velox from Source."with ccache hit rates above 92%.Normal path (Stash hit)
The normal Stash-first path was validated and merged in #12610. On an earlier revision of that PR,
build-native-lib-centos-8 (ARM64)showed:stash-hit=trueRelationship to other PRs
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude