Spark can't read its own write after another one commits data to the iceberg table

[chenzl25]

Apache Iceberg version

1.5.2

Query engine

Spark

Please describe the bug 🐞

How to reproduce:

• Spark-sql runs CREATE TABLE demo.demo_db.t_bug(a int) TBLPROPERTIES ('format-version'='2')
• Spark-sql runs insert into demo.demo_db.t_bug values (1)
• Another SDK (icelake): insert into demo.demo_db.t_bug values (2)
• After that, the weird thing happens
• select * from demo.demo_db.t_bug.files told me there are 2 parquet files, but when I run select * from demo.demo_db.t_bug only 1 row returns. And I run 2 time travel queries, you can see both snapshots return 1 row. The first time travel query result with id 6579104674932036030 is expected, but the second one with id 6579104674932036031 is unexpected

• BTW, some query engines like clickhouse and duckdb can read 2 rows, so I think something wrong in spark.

I also include the table t_bug metadata and data directory here for those who are interested in this issue.

t_bug.zip

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[lurnagao-dahua]

What is the result of executing query using SparkSQL:
select * from demo.demo_db.t_bug

[chenzl25]

Never mind. I found it is a bug of icelake. It doesn't follow the iceberg spec. icelake-io/icelake#280 Let me close this issue.