Description
Spark 4.2.0 was released on July 14, 2026. Gluten should add initial Spark 4.2.x support, following the Spark 4.1.x support process in #11339.
Spark 4.2 expands Spark SQL/data types, Data Source V2, CDC, Arrow/PySpark behavior, and streaming/state-store areas. These should be validated during the bring-up for possible shim, compatibility, test, or follow-up offload work.
Proposed work
- Fix compatibility issues introduced by Spark 4.2
- Add Spark 4.2.x build profile
- Add Spark 4.2.x shim layer
- Add Spark 4.2.x unit test module
- Add/enable Spark 4.2.x test suites in CI
- Track Spark 4.2.x failed unit tests
Validation areas
- new geospatial types/functions
- DSv2 CDC, transactions, schema evolution, and metrics
- Arrow/PySpark default behavior changes
- streaming and state-store changes
- new SQL functions, vector/sketch functions, and path resolution
Goal
Initial support should focus on build, shim, and unit-test compatibility first. Native offload support for new Spark 4.2 features can be handled in follow-up issues.
Description
Spark 4.2.0 was released on July 14, 2026. Gluten should add initial Spark 4.2.x support, following the Spark 4.1.x support process in #11339.
Spark 4.2 expands Spark SQL/data types, Data Source V2, CDC, Arrow/PySpark behavior, and streaming/state-store areas. These should be validated during the bring-up for possible shim, compatibility, test, or follow-up offload work.
Proposed work
Validation areas
Goal
Initial support should focus on build, shim, and unit-test compatibility first. Native offload support for new Spark 4.2 features can be handled in follow-up issues.