[SPARK-56985] Upgrade Iceberg example to use Iceberg 1.11.0

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade Iceberg to 1.11.0 in the spark-connect-server-iceberg.yaml example.

Why are the changes needed?

To use the latest Iceberg release.

• https://iceberg.apache.org/releases/#1110-release
• https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.11.0 (2026-05-19)
  • Core: Drop support for Java 11 iceberg#14400
  • Spark 3.4: Deprecate support iceberg#14099
  • Spark: Add support for Spark 4.1 iceberg#14155

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual review.

$ kubectl apply -f examples/spark-connect-server-iceberg.yaml                                                    
sparkapplication.spark.apache.org/spark-connect-server-iceberg created

$ kubectl get sparkapp
NAME                           AGE   CURRENT STATE
spark-connect-server-iceberg   46s   RunningHealthy

$ kybectl get pod                                            
NAME                                         READY   STATUS    RESTARTS      AGE
spark-connect-server-iceberg-0-driver        1/1     Running   0             47s
spark-connect-server-iceberg-exec-1          1/1     Running   0             11s
spark-connect-server-iceberg-exec-2          1/1     Running   0             11s
spark-connect-server-iceberg-exec-3          1/1     Running   0             11s
spark-kubernetes-operator-86bfb88655-w5fs8   1/1     Running   7 (42h ago)   6d19h

$ kubectl logs -f spark-connect-server-iceberg-0-driver
...
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13 added as a dependency
...
        found org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0 in central
...
downloading https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.0_2.13/1.11.0/iceberg-spark-runtime-4.0_2.13-1.11.0.jar ...
        [SUCCESSFUL ] org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0!iceberg-spark-runtime-4.0_2.13.jar (834ms)
...
        org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0 from central in [default]
...
Files file:///tmp/.ivy2.5.2/jars/org.apache.iceberg_iceberg-spark-runtime-4.0_2.13-1.11.0.jar from /tmp/.ivy2.5.2/jars/org.apache.iceberg_iceberg-spark-runtime-4.0_2.13-1.11.0.jar to /opt/spark/work-dir/org.apache.iceberg_iceberg-spark-runtime-4.0_2.13-1.11.0.jar
...
26/05/21 14:37:11 INFO SparkConnectServer: Starting Spark session.
26/05/21 14:37:11 INFO SparkContext: Running Spark version 4.0.2
26/05/21 14:37:11 INFO SparkContext: OS info Linux, 6.6.119-0-virt, aarch64
26/05/21 14:37:11 INFO SparkContext: Java version 17.0.17
...
26/05/21 14:37:17 INFO Utils: Successfully started service 'org.apache.spark.sql.connect.service.SparkConnectService$' on port 15002.
26/05/21 14:37:17 INFO SparkConnectServer: Spark Connect server started at: 0:0:0:0:0:0:0:0:15002

$ kubectl port-forward spark-connect-server-iceberg-0-driver 15002

$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.2
      /_/

Type in expressions to have them evaluated.
Spark connect server version 4.0.2.
Spark session available as 'spark'.

scala>
sql("""CREATE TABLE taxis(vendor_id bigint, trip_id bigint, trip_distance float, fare_amount double, store_and_fwd_flag string) PARTITIONED BY (vendor_id);""").show()
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
++
||
++
++

scala>
sql("""INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');""").show()
++
||
++
++

scala> sql("SELECT * FROM taxis").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1        |1000374|8.4          |42.13      |Y                 |
|1        |1000371|1.8          |15.32      |N                 |
|2        |1000373|0.9          |9.01       |N                 |
|2        |1000372|2.5          |22.15      |N                 |
+---------+-------+-------------+-----------+------------------+

scala> sql("SELECT * FROM taxis.history").show(false)
+-----------------------+-------------------+---------+-------------------+
|made_current_at        |snapshot_id        |parent_id|is_current_ancestor|
+-----------------------+-------------------+---------+-------------------+
|2026-05-21 14:55:46.629|4314466511351517368|NULL     |true               |
+-----------------------+-------------------+---------+-------------------+

scala> sql("SELECT * FROM taxis VERSION AS OF 4314466511351517368").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1        |1000374|8.4          |42.13      |Y                 |
|1        |1000371|1.8          |15.32      |N                 |
|2        |1000373|0.9          |9.01       |N                 |
|2        |1000372|2.5          |22.15      |N                 |
+---------+-------+-------------+-----------+------------------+

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.7

[dongjoon-hyun]

Thank you, @viirya !

[dongjoon-hyun]

Merged to main