Skip to content

[lake/tiering] Tiering source and committer get chained into one task when source parallelism is 1 #3801

Description

@beryllw

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

In the lake tiering job built by LakeTieringJobBuilder, the TieringCommitter operator has parallelism 1. When the TieringSource also runs with parallelism 1 (common for small deployments), Flink chains the source and the committer into a single task, since they have the same parallelism and a FORWARD edge.

This causes problems:

  1. Blocking: the source (heavy IO: reading Fluss data and writing lake files) and the committer (slow lake commit + coordinator RPC) share one task thread. A slow commit blocks the tiering write loop and delays checkpoint barriers.
  2. Inconsistent topology: with source parallelism > 1 they are never chained (parallelism mismatch), so the runtime behavior changes unexpectedly when scaling down to 1.
  3. Poor observability: busy/backpressure metrics of the two operators are merged, making it hard to tell whether writing or committing is the bottleneck.

Solution

Call startNewChain() on the TieringCommitter operator so it never chains with the source, regardless of parallelism. The committer stays chained with the trailing DiscardingSink. This is the same mechanism Paimon uses for its writer/committer separation (sink.committer-operator-chaining).

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions