Search before asking
Fluss version
main (development)
Please describe the bug 🐞
In the lake tiering job built by LakeTieringJobBuilder, the TieringCommitter operator has parallelism 1. When the TieringSource also runs with parallelism 1 (common for small deployments), Flink chains the source and the committer into a single task, since they have the same parallelism and a FORWARD edge.
This causes problems:
- Blocking: the source (heavy IO: reading Fluss data and writing lake files) and the committer (slow lake commit + coordinator RPC) share one task thread. A slow commit blocks the tiering write loop and delays checkpoint barriers.
- Inconsistent topology: with source parallelism > 1 they are never chained (parallelism mismatch), so the runtime behavior changes unexpectedly when scaling down to 1.
- Poor observability: busy/backpressure metrics of the two operators are merged, making it hard to tell whether writing or committing is the bottleneck.
Solution
Call startNewChain() on the TieringCommitter operator so it never chains with the source, regardless of parallelism. The committer stays chained with the trailing DiscardingSink. This is the same mechanism Paimon uses for its writer/committer separation (sink.committer-operator-chaining).
Are you willing to submit a PR?
Search before asking
Fluss version
main (development)
Please describe the bug 🐞
In the lake tiering job built by
LakeTieringJobBuilder, theTieringCommitteroperator has parallelism 1. When theTieringSourcealso runs with parallelism 1 (common for small deployments), Flink chains the source and the committer into a single task, since they have the same parallelism and a FORWARD edge.This causes problems:
Solution
Call
startNewChain()on theTieringCommitteroperator so it never chains with the source, regardless of parallelism. The committer stays chained with the trailingDiscardingSink. This is the same mechanism Paimon uses for its writer/committer separation (sink.committer-operator-chaining).Are you willing to submit a PR?