Add spark 4.0.0-preview2 support

[jbonofre]

Feature Request / Improvement

In order to avoid to be in a rush, I propose to start Spark 4.0.0 support by adding 4.0.0-preview2 support.

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[ajantha-bhat]

@huaxingao started it long back and PR got closed due to inactivity
#11583

[jbonofre]

@ajantha-bhat yes I saw that, and also we had another GH issue about that. As it seems that Spark 4.0.0 is coming at some point, I think it's worth to resume what @huaxingao started but with the preview2 "release".

[huaxingao]

@ajantha-bhat @jbonofre I stopped the 4.0.0-preview2 integration because there was a bug in 4.0.0-preview2 affecting Iceberg, which was fixed later. I resumed the 4.0 work yesterday and hope to finish it soon.