Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog by jackylee-ch · Pull Request #15814 · apache/iceberg

jackylee-ch · 2026-03-28T09:07:37Z

When using a Hive catalog, schema evolution operations that change column positions — such as ALTER TABLE ... DROP COLUMN (non-last column) and ALTER TABLE ... ALTER COLUMN ... FIRST/AFTER (reorder) — fail with InvalidOperationException from MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange. This is because the Hive Metastore validates schema changes by comparing column types positionally (by index, not by name), controlled by hive.metastore.disallow.incompatible.col.type.changes (default true).

This limitation is not documented anywhere in the Iceberg docs, though the Iceberg test suite itself works around it by setting METASTORE_DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES=false (TestHiveMetastore.java:269), and a test comment(HiveTableTest.java:283-287) explicitly describes the issue.

This PR adds !!! warning admonition blocks to:

spark-ddl.md — under the DROP COLUMN section (full explanation, workaround, and trade-off) and the ALTER COLUMN reorder section (with cross-reference)
flink-ddl.md — under the Hive catalog configuration section (engine-agnostic warning)

…nd REORDER When using a Hive catalog, ALTER TABLE DROP COLUMN (non-last column) and ALTER COLUMN REORDER fail because the Hive Metastore validates schema changes by comparing column types positionally. Dropping a middle column shifts subsequent columns, causing HMS to reject the change as an incompatible type change via MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange. Add warning admonitions to spark-ddl.md (DROP COLUMN and REORDER sections) and flink-ddl.md (Hive catalog section) documenting the limitation, workaround (hive.metastore.disallow.incompatible.col.type.changes=false), and trade-off (Hive engine can no longer read the table).

pan3793 · 2026-03-29T16:34:38Z

docs/docs/spark-ddl.md

+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.


can not recall the detail, but I remember that overriding this config on the client-side does not work on some HMS versions.

Hm, I have tried it for hive 2.3.9 and 3.1.3. Do you remember the HMS version you tried that failed?

did you test with embedded HMS, or remote HMS? IIRC, Hive 2.3.9 with remote HMS does not work. I will contact our Hive team and sync the result today.

I was testing with embedded HMS. I just asks the colleague in charge of Hive, these should be configured in the metadata store, unless user use embedded HMS just like I did.

I would update the doc, thanks.

back here, I get a response that config hive.metastore.disallow.incompatible.col.type.changes client-side overriding requires HIVE-17832, HIVE-17942

I have check remote HMS with version 3.1.3, which contains the HIVE-17832 and HIVE-17942, they still don't work. Thus, I updated the documentation as follows.
- Remote HMS: Set this property in the HMS server's hive-site.xml.
- Embedded HMS: Add the equivalent property to the Hive catalog configuration.

you should apply these patches to the HMS client ...

you should apply these patches to the HMS client ...

Oh. got it, would make a double check for hive client 3.x

I have test with flink 1.20 and hive client 3.1.3, it doesn't work.

The hms client also needs to actively use setMetaConf to make it effective. However, neither Spark, Flink, nor Iceberg currently offer this type of operation, so configuring it directly in the job will not work.

@jackylee-ch, I see, and have double-checked with our hive team, there are additional internal changes to make client-side overriding work. Sorry about the incomplete information.

wypoon

It may be better to add the warning in the ALTER TABLE section before any of the ALTER TABLE ... XXX commands.
There you can call out the ALTER TABLE ... ADD COLUMN, ALTER TABLE ... ALTER COLUMN, and ALTER TABLE ... DROP COLUMN commands, if they change the ordinal position of existing columns.
This way, the top level explanation and the workaround is presented first. Then under the individual examples, you can point them out as changing the ordinal position of existing columns (without needing to explain the workaround).

docs/docs/spark-ddl.md

jackylee-ch · 2026-04-03T06:54:39Z

cc @manuzhang @huaxingao

huaxingao · 2026-04-03T16:55:37Z

LGTM. I will leave the PR open for a couple of more days in case @manuzhang or other want to take a look.

wypoon

This is better now. Thanks.

Copilot

Pull request overview

Adds documentation warnings about Hive Metastore (HMS) schema-evolution limitations when using an Iceberg Hive catalog, specifically for schema changes that shift column positions and can fail due to HMS positional validation.

Changes:

Added a detailed “Hive Catalog Limitation” warning to Spark DDL docs, plus shorter per-operation warnings for ADD COLUMN ... FIRST/AFTER, ALTER COLUMN ... FIRST/AFTER, and DROP COLUMN (non-last).
Added an engine-agnostic “Hive Catalog Limitation” warning to the Flink DDL Hive catalog configuration section, including a workaround and trade-off explanation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
docs/docs/spark-ddl.md	Adds warning admonitions describing HMS positional schema validation issues and suggested workaround/trade-offs for Spark DDL schema evolution.
docs/docs/flink-ddl.md	Adds a Hive catalog limitation warning in Flink DDL docs to make the HMS behavior and workaround visible in Flink context.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/docs/spark-ddl.md

docs/docs/flink-ddl.md

Copilot · 2026-04-04T16:16:52Z

docs/docs/flink-ddl.md

+    `hive.metastore.disallow.incompatible.col.type.changes=false`:
+
+    - **Remote HMS:** Set this property in the HMS server's `hive-site.xml`.
+    - **Embedded HMS:** Add the equivalent property to the Hive catalog configuration.


The “Embedded HMS” workaround is unclear in the context of this Flink Hive catalog section (where uri is required and configuration is typically provided via hive-conf-dir/classpath). Consider rephrasing this to explicitly instruct users to set the property in a hive-site.xml picked up via hive-conf-dir (or classpath) rather than suggesting an “embedded” metastore configuration path that may not apply.

Suggested change

- **Embedded HMS:** Add the equivalent property to the Hive catalog configuration.

- **When configuring the Hive catalog in Flink:** Set this property in a `hive-site.xml`

that Flink picks up via `hive-conf-dir` or from the classpath.

The Embedded HMS means using HMS with derby, which is launching in the Flink JM.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions bot added the docs label Mar 28, 2026

pan3793 reviewed Mar 29, 2026

View reviewed changes

Docs: Clarify HMS workaround for embedded vs remote deployment

8ed9905

wypoon reviewed Mar 31, 2026

View reviewed changes

docs/docs/spark-ddl.md Show resolved Hide resolved

Docs: add more warning for spark-ddl.md

7c8417f

huaxingao approved these changes Apr 3, 2026

View reviewed changes

wypoon approved these changes Apr 3, 2026

View reviewed changes

pan3793 approved these changes Apr 4, 2026

View reviewed changes

manuzhang requested a review from Copilot April 4, 2026 16:14

Copilot started reviewing on behalf of manuzhang April 4, 2026 16:15 View session

Copilot AI reviewed Apr 4, 2026

View reviewed changes

Apply suggestions from code review

6d1a400

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

	- Embedded HMS: Add the equivalent property to the Hive catalog configuration.
	- When configuring the Hive catalog in Flink: Set this property in a `hive-site.xml`
	that Flink picks up via `hive-conf-dir` or from the classpath.

Conversation

jackylee-ch commented Mar 28, 2026

Uh oh!

pan3793 Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wypoon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jackylee-ch commented Apr 3, 2026

Uh oh!

huaxingao commented Apr 3, 2026

Uh oh!

wypoon left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pan3793 Mar 29, 2026 •

edited

Loading