Skip to content

docs: troubleshoot iceberg commit error when adding column #786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sweb merged 1 commit into from
Oct 23, 2026
Merged

docs: troubleshoot iceberg commit error when adding column #786

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/modules/hive/pages/troubleshooting/index.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,51 @@ spec:
----

NOTE: This will suppress all logging from TThreadPoolServer, including log events that might be useful for diagnosing issues.

== Iceberg schema changes fail with 'types incompatible with the existing columns'

Changing the schema of an Apache Iceberg table that uses the Hive metastore as its catalog fails when the change moves existing columns to a different position.
Examples are adding a column with `FIRST` or `AFTER`, or dropping a column that is not the last one:

[source,sql]
----
CREATE TABLE iceberg.demo.t (id BIGINT, name VARCHAR);
ALTER TABLE iceberg.demo.t ADD COLUMN c0 INTEGER FIRST;
----

Trino reports:

[source]
----
ICEBERG_COMMIT_ERROR: Failed to add column: The following columns have types incompatible with the existing columns in their respective positions :
c0,id
----

The metastore logs the corresponding error:

[source]
----
ERROR [Metastore-Handler-Pool: Thread-115] metastore.RetryingHMSHandler: InvalidOperationException(message:The following columns have types incompatible with the existing columns in their respective positions :
c0,id)
at org.apache.hadoop.hive.metastore.DefaultIncompatibleTableChangeHandler.checkColTypeChangeCompatible(DefaultIncompatibleTableChangeHandler.java:112)
----

Iceberg tracks columns by ID and supports these changes.
On every commit, Iceberg also writes the new column list to the table in the Hive metastore.
The metastore compares the old and the new column list position by position and rejects the update if the type at a position changed in a way Hive considers incompatible, for example from `string` to `bigint`.
Appending a column at the end does not trigger the check.
The rejected commit leaves the Iceberg table unchanged, so the statement can be repeated once the check is disabled.

Disable the check with a config override:

[source,yaml]
----
spec:
metastore:
configOverrides:
hive-site.xml:
hive.metastore.disallow.incompatible.col.type.changes: "false"
----

NOTE: The setting applies to all tables in the metastore, including plain Hive tables.
For those, the check is relevant as it prevents schema changes that make existing data unreadable in formats that map columns by position, such as CSV.
Loading