Description
Apache Flink has released version 2.x series (2.0, 2.1, 2.2), but StreamPark currently only supports up to Flink 1.20. This feature request is to add support for Flink 2.x versions.
Key Changes in Flink 2.x
- Removed Scala DataStream API:
org.apache.flink.streaming.api.scala._ and org.apache.flink.table.api.bridge.scala._ are completely removed
- ParameterTool relocated: Moved from
org.apache.flink.api.java.utils.ParameterTool to org.apache.flink.util.ParameterTool
- New APIs: Model API (2.1+), FunctionDescriptor (2.2+), Materialized Tables (2.2+)
- Deprecated APIs removed: DataSet API, SourceFunction/SinkFunction, Per-Job deployment mode
Implementation Approach
- Version Detection: Extend
FlinkVersion.scala to recognize 2.x versions
- New Shims Modules: Create
streampark-flink-shims_flink-2.0, streampark-flink-shims_flink-2.1, streampark-flink-shims_flink-2.2
- Java API Migration: Use Java DataStream API instead of removed Scala API
- Self-contained 2.x Modules: Each 2.x module includes its own base classes with corrected
ParameterTool imports
- Console Service Integration: Update
EnvInitializer.java and pom.xml to recognize 2.x shims JARs
Files Modified
streampark-common/.../FlinkVersion.scala - Version detection
streampark-flink-shims/pom.xml - Add 2.0/2.1/2.2 modules
streampark-flink-shims-base/... - Add V2 compatible classes
streampark-flink-shims_flink-2.0/** - New module (19 source files)
streampark-flink-shims_flink-2.1/** - New module (19 source files + Model API)
streampark-flink-shims_flink-2.2/** - New module (19 source files + Model/FuncDesc API)
console-service/pom.xml - Add 2.0/2.1/2.2 shims JAR dependencies
console-service/.../EnvInitializer.java - Update regex to match 2.x
Testing
- All shims modules compile successfully (1.12-1.20 + 2.0-2.2)
- Unit tests pass
- Console service compiles successfully
Usage Scenario
Users can now deploy and manage Flink 2.x applications through StreamPark, taking advantage of Flink 2.x new features like decoupled state management, async state execution, and enhanced SQL capabilities.
Related issues
None
Are you willing to submit a PR?
Yes I am willing to submit a PR!
Code of Conduct
I agree to follow this project's Code of Conduct
Description
Apache Flink has released version 2.x series (2.0, 2.1, 2.2), but StreamPark currently only supports up to Flink 1.20. This feature request is to add support for Flink 2.x versions.
Key Changes in Flink 2.x
org.apache.flink.streaming.api.scala._andorg.apache.flink.table.api.bridge.scala._are completely removedorg.apache.flink.api.java.utils.ParameterTooltoorg.apache.flink.util.ParameterToolImplementation Approach
FlinkVersion.scalato recognize 2.x versionsstreampark-flink-shims_flink-2.0,streampark-flink-shims_flink-2.1,streampark-flink-shims_flink-2.2ParameterToolimportsEnvInitializer.javaandpom.xmlto recognize 2.x shims JARsFiles Modified
streampark-common/.../FlinkVersion.scala- Version detectionstreampark-flink-shims/pom.xml- Add 2.0/2.1/2.2 modulesstreampark-flink-shims-base/...- Add V2 compatible classesstreampark-flink-shims_flink-2.0/**- New module (19 source files)streampark-flink-shims_flink-2.1/**- New module (19 source files + Model API)streampark-flink-shims_flink-2.2/**- New module (19 source files + Model/FuncDesc API)console-service/pom.xml- Add 2.0/2.1/2.2 shims JAR dependenciesconsole-service/.../EnvInitializer.java- Update regex to match 2.xTesting
Usage Scenario
Users can now deploy and manage Flink 2.x applications through StreamPark, taking advantage of Flink 2.x new features like decoupled state management, async state execution, and enhanced SQL capabilities.
Related issues
None
Are you willing to submit a PR?
Yes I am willing to submit a PR!
Code of Conduct
I agree to follow this project's Code of Conduct