Apache Iceberg Scheme Evolution using Spark

Question

Currently I am using Iceberg in my project, so I am having one doubt in that.

My Current Scenario:

1. I have loaded the data into my Iceberg table using spark data frame(this is my doing through spark job)

   df.writeTo("catalog.mydb.test2").using("iceberg").create()
   

2. Now From source side I have added two colums and started the Job which is doing merge

   df.createOrReplaceTempView("myview")
   spark.sql("MERGE INTO catalog.mydb.test2 as t USING (SELECT * FROM myview) as s ON t.id = s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT ")
   

Doing both of these step I am expecting new columns to be added into the target table but it did not work.

As I can see Iceberg Support full schema evolution.. What does it means..if it is not adding any columns dynamically to my target table.

Please help how can I achieve adding new columns into my target table dynmically.

Answer 1

You can enable this with merge-schema option, but we don't recommend it because, as @shay__ points out, it can sometimes cause unmanageable catastrophes.