Skip to content

[Feature] Support Apache Flink 2.x (2.0, 2.1, 2.2) #4367

Description

@shangeyao

Description

Apache Flink has released version 2.x series (2.0, 2.1, 2.2), but StreamPark currently only supports up to Flink 1.20. This feature request is to add support for Flink 2.x versions.

Key Changes in Flink 2.x

  • Removed Scala DataStream API: org.apache.flink.streaming.api.scala._ and org.apache.flink.table.api.bridge.scala._ are completely removed
  • ParameterTool relocated: Moved from org.apache.flink.api.java.utils.ParameterTool to org.apache.flink.util.ParameterTool
  • New APIs: Model API (2.1+), FunctionDescriptor (2.2+), Materialized Tables (2.2+)
  • Deprecated APIs removed: DataSet API, SourceFunction/SinkFunction, Per-Job deployment mode

Implementation Approach

  1. Version Detection: Extend FlinkVersion.scala to recognize 2.x versions
  2. New Shims Modules: Create streampark-flink-shims_flink-2.0, streampark-flink-shims_flink-2.1, streampark-flink-shims_flink-2.2
  3. Java API Migration: Use Java DataStream API instead of removed Scala API
  4. Self-contained 2.x Modules: Each 2.x module includes its own base classes with corrected ParameterTool imports
  5. Console Service Integration: Update EnvInitializer.java and pom.xml to recognize 2.x shims JARs

Files Modified

  • streampark-common/.../FlinkVersion.scala - Version detection
  • streampark-flink-shims/pom.xml - Add 2.0/2.1/2.2 modules
  • streampark-flink-shims-base/... - Add V2 compatible classes
  • streampark-flink-shims_flink-2.0/** - New module (19 source files)
  • streampark-flink-shims_flink-2.1/** - New module (19 source files + Model API)
  • streampark-flink-shims_flink-2.2/** - New module (19 source files + Model/FuncDesc API)
  • console-service/pom.xml - Add 2.0/2.1/2.2 shims JAR dependencies
  • console-service/.../EnvInitializer.java - Update regex to match 2.x

Testing

  • All shims modules compile successfully (1.12-1.20 + 2.0-2.2)
  • Unit tests pass
  • Console service compiles successfully

Usage Scenario

Users can now deploy and manage Flink 2.x applications through StreamPark, taking advantage of Flink 2.x new features like decoupled state management, async state execution, and enhanced SQL capabilities.

Related issues

None

Are you willing to submit a PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions