Core,Api: Add overwrite option when register external table to catalog

[dramaticlly]

This PR adds a new register-table with overwrite option on Catalog interface to allow overwrite table metadata of an existing Iceberg table. The overwrite is achieved via TableOperations.commit(base, new) for catalogs extends BaseMetastoreCatalog.

• Relate to RESTTableOperations does not support table metadata swap like others TableOperations did #12134
• openAPI REST spec change merged in OpenAPI: Add overwrite option when registering an iceberg table #12239

[dramaticlly]

Java CI Failure is timing out on concurrent fast append and seems unrelated to the change.

@rdblue @RussellSpitzer @danielcweeks do you want to take a look?

[gaborkaszab]

Thanks for working on this, @dramaticlly !

[RussellSpitzer]

Are we just adding a bucket to test the change? Why not just use the table UUID? I feel like we should be able to just

Make Table 1
Make Table 2
Register overwrite Table1 with Table2
Check that metadata table1 matches table 2?

[dramaticlly]

Initially I think register with overwrite helps revert an existing table to a new previous health state. If we want to support overwrite with another tables's metadata, It seems better suited with drop + register, to reflect the table UUID change.

From the table spec, it asks Implementations to throw an exception if a table's UUID does not match the expected UUID when refreshing metadata. What do you think?

[RussellSpitzer]

Should be a second test I think?

[RussellSpitzer]

Second test?

[RussellSpitzer]

I'm a little worried about passing through current metadata here. Is this just a workaround to the normal commit logic?

If the metadata changes from "current" by the time an overwrite request goes through then we don't want a retry or failure, it should still pass?

[dramaticlly]

I was hoping to reuse the existing commit logic for atomicity support, and also better lineage to track previous/old metadata for hive table and JDBC tables.

As for potential conflict when base is out of date, I think that's a valid concern and we probably do not want this operation fail as user intent is replace with provided table metadata. I am thinking about add a retry block to help, please let me know if you feel otherwise

AtomicBoolean isRetry = new AtomicBoolean(false);
// commit with retry
Tasks.foreach(ops)
  .retry(COMMIT_NUM_RETRIES_DEFAULT)
  .exponentialBackoff(
      COMMIT_MIN_RETRY_WAIT_MS_DEFAULT,
      COMMIT_MAX_RETRY_WAIT_MS_DEFAULT,
      COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT,
      2.0 /* exponential */)
  .onlyRetryOn(CommitFailedException.class)
  .run(
      taskOps -> {
        TableMetadata base = isRetry.get() ? taskOps.refresh() : taskOps.current();
        isRetry.set(true);
        taskOps.commit(base, newMetadata);
      });