Add option to provide partition spec in spark ADD_FILES procedure

[bharos]

Feature Request / Improvement

Currently, the ADD_FILES API in Apache Iceberg does not support specifying a partition spec, meaning that the API always operates on the latest table spec when adding files, as shown in the implementation.

This can become problematic when the table or partition spec has evolved over time. For instance, in an archival and restore tool where data was archived before the partition spec changed, it would be beneficial to restore archived data using the older partition spec, rather than the current one.

I am working on a PR to address this and would appreciate any specific suggestions or concerns from the community on making this change.

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[RussellSpitzer]

Please see #12319

[bharos]

@RussellSpitzer thanks, I see your change addresses the spec finding in SparkTableUtil.importSparkTable method

I just created the PR for adding the spec as argument to add_files here #12327

Can you review my PR as I find that this change is still needed, to allow passing spec for the FileTable usecase.
I am adding the partition_spec_version option to be only used for the FileTable and it will be a no-op for the SourceTable case

[RussellSpitzer]

I think I want to do something similar here, Instead of passing in the partition spec can we just search the Iceberg table to see if a valid spec exists that matches the FileTable?

[bharos]

Makes sense to avoid passing the argument if possible. I'll update the PR