[Bug] Table stuck in WAITING_STABLE after cancelling a schema-change job that never became stable

[Baymine]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

When a schema-change job is cancelled while the table is still in the WAITING_STABLE sub-state (the table never became stable, so the job was never promoted to SCHEMA_CHANGE), the table's OlapTableState is not reset back to NORMAL.

Root cause: SchemaChangeJobV2.changeTableState (invoked from the job's cancelInternal) only resets the table when its state is SCHEMA_CHANGE; a table in WAITING_STABLE is left untouched. In addition, SchemaChangeHandler.cancelColumnJob throws "could not find related job" when a table is flagged WAITING_STABLE but has no live job (an orphaned state older binaries could leave behind).

As a result the table stays permanently stuck in WAITING_STABLE with no live job, and every subsequent ALTER / TRUNCATE / DROP PARTITION / RENAME / CANCEL ALTER on it is rejected — the only way to recover is DROP TABLE.

What You Expected?

Cancelling a schema-change job that is still WAITING_STABLE should reset the table back to NORMAL so it is usable again. A table left in an orphaned WAITING_STABLE (no backing job) should be self-healed by CANCEL ALTER TABLE rather than requiring DROP TABLE.

How to Reproduce?

1. Start an ALTER TABLE ... ADD COLUMN on a table that is not stable (e.g. a replica in DECOMMISSION), so the pending schema-change job parks in WAITING_STABLE.
2. CANCEL ALTER TABLE COLUMN FROM db.tbl.
3. The table remains in WAITING_STABLE; subsequent ALTER/TRUNCATE/DROP PARTITION/RENAME/CANCEL are rejected.

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_e7b2388728fd
This content is generated by AI for reference only.

Initial assessment: valid, high-confidence FE metadata-state bug.

I checked the live issue metadata and the relevant code on current upstream master (2215dc476a1f9a1f6edba8c7a056e450165f7001). The issue is open and currently has no labels, assignee, milestone, or comments. PR #66178 is linked and open; it has no reviews yet and its CI is still pending. The affected files are unchanged between upstream master and the local checkout used for inspection.

The reported state transition is deterministic:

• A schema-change job is created in PENDING while the table is initially set to SCHEMA_CHANGE.
• AlterJobV2.checkTableStable() changes the table to WAITING_STABLE and returns false when the table is unstable; the job itself remains PENDING.
• SchemaChangeJobV2.cancelImpl() runs cancelInternal(), then calls changeTableState(..., NORMAL). That helper only performs the transition when the current table state is SCHEMA_CHANGE; it silently ignores WAITING_STABLE.
• The job is nevertheless marked CANCELLED and journaled. The first CANCEL ALTER TABLE COLUMN therefore leaves the table in WAITING_STABLE with no unfinished schema-change job.
• A subsequent cancel reaches SchemaChangeHandler.cancelColumnJob(), finds no unfinished job, and throws the reported “could not find related job” error. OlapTable.checkNormalStateForAlter() then rejects later ALTER, partition, rename, and truncate paths while the state remains non-NORMAL.

Git history supports treating this as a regression. WAITING_STABLE was introduced as a temporary repair-permitting state, and cancellation while unstable was explicitly enabled later. A subsequent refactor moved table-state cleanup into SchemaChangeJobV2; the new helper retained only the SCHEMA_CHANGE predicate and omitted WAITING_STABLE.

Linked PR assessment: the direction is correct, but #66178 needs an ownership guard before merge.

• The PR's orphan branch checks only SchemaChangeHandler.getUnfinishedAlterJobV2ByTableId(). Because WAITING_STABLE is also used by RollupJobV2, CANCEL ... COLUMN against a table with a live unstable rollup job would find no schema-change job and durably force the table to NORMAL via setTableStatusInternal(). The rollup job is still live. The recovery path must also check MaterializedViewHandler (and preserve the state if any backing alter job exists).
• The PR broadens the generic SchemaChangeJobV2.changeTableState() helper, but that helper is called after normal finish, cancel, replay-finish, and replay-cancel—not only the cancellation path. Allowing every caller to reset WAITING_STABLE can let a stale/replayed schema-change job reset a WAITING_STABLE currently owned by a rollup job. Limit the new source-state transition to cancellation/replay-cancellation with an ownership invariant, or make callers supply the expected source states.
• The PR releases the table lock before calling setTableStatusInternal(), which performs an unconditional state change under a newly acquired lock. The durable recovery operation should re-check that the table is still WAITING_STABLE and still has no backing schema-change or rollup job at the point of mutation.

Recommended fix/review direction:

1. Make cancellation of the live schema-change job transition its own table from WAITING_STABLE to NORMAL, without broadening unrelated finish/replay paths.
2. Self-heal only a genuinely orphaned WAITING_STABLE: verify that neither the schema-change handler nor the materialized-view/rollup handler has an unfinished job, and revalidate the state atomically when applying the durable reset.
3. Reconcile persistence semantics. OlapTable documents WAITING_STABLE as in-memory-only, but the state field is Gson-serialized and no WAITING_STABLE normalization exists in gsonPostProcess() or elsewhere in the current source. An image can therefore retain an orphaned state. Either enforce the intended non-persistence or reconcile table states against loaded alter jobs after metadata load.

Required tests:

• Extend SchemaChangeJobV2Test.testSchemaChangeWhileTabletNotStable: after forcing a replica to DECOMMISSION, assert table state WAITING_STABLE, cancel through SchemaChangeHandler (the user-facing path), and assert job state CANCELLED plus table state NORMAL. PR [fix](schema-change) reset table state when cancelling a job stuck in WAITING_STABLE #66178 currently calls schemaChangeJob.cancel() directly.
• Set a table to WAITING_STABLE with no unfinished alter job, run CANCEL ALTER TABLE COLUMN, and assert successful recovery to NORMAL.
• Keep an unfinished rollup/MV job for a WAITING_STABLE table and verify the schema-change orphan-recovery path does not reset the table.
• Add cancelled-job replay/image-load coverage so WAITING_STABLE cannot survive without a backing job.

No additional logs, runtime profile, or reproduction details are required to confirm the bug on master; the current code establishes it directly. For deciding release-branch backports, the first affected release/commit should still be identified from branch history.