Support FIRST, AFTER and LAST clause when adding a new column in engine and Iceberg connector

[ebyhr]

Description

I propose adding FIRST, AFTER and LAST options to ALTER TABLE ... ADD COLUMN statement so that we can add new columns and fields to specific positions. LAST is the default when the option isn't provided.

    | ALTER TABLE (IF EXISTS)? tableName=qualifiedName
        ADD COLUMN (IF NOT EXISTS)? column=columnDefinition
        (FIRST | LAST | AFTER after=identifier)? 

Column case:

CREATE TABLE catalog.schema.table(b INTEGER)
ALTER TABLE catalog.schema.table ADD COLUMN a INTEGER FIRST COMMENT 'x' WITH (p = v)
(b INTEGER)
↓
(a INTEGER, b INTEGER)

CREATE TABLE catalog.schema.table(a INTEGER, c INTEGER)
ALTER TABLE catalog.schema.table ADD COLUMN b INTEGER AFTER a COMMENT 'x' WITH (p = v)
(a INTEGER, c INTEGER)
↓
(a INTEGER, b INTEGER, c INTEGER)

Datasources supporting these options:

• Spark https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table-add-column
• Hive https://cwiki.apache.org/confluence/display/hive/languagemanual+ddl
• MySQL https://dev.mysql.com/doc/refman/8.3/en/alter-table.html
• ClickHouse https://clickhouse.com/docs/en/sql-reference/statements/alter/column#add-column

Fixes #20091

Release notes

(x) Release notes are required, with the following suggested text:

# General, Iceberg
* Add support `FIRST`, `AFTER` and `LAST` clause when adding a new column. ({issue}`20091`)

[findinpath]

From a Zoom Meeting with @martint , @electrum

Support adding a field with ADD COLUMN in Iceberg
#16321

It would be nice to support adding nested fields in ROW types.

Positioning of the fields seems also attractive (FIRST/ AFTER column), but it is not in the spec.
Reference https://iceberg.apache.org/docs/latest/spark-ddl/
Alternative scenario in Trino for specifying the exact position of the new nested field in the row: ALTER COLUMN ... SET DATA TYPE

FIRST/AFTER - let’’s wait for request from the community to actually start any work on this area.

[mosabua]

If we add FIRST .. why not also add LAST?

[mosabua]

Also .. is there any chance we implement this later for JDBC connectors and others across the board?

[martint]

The single-level scenario syntax looks good.

The nested case is somewhat confusing. At first glance, it's not clear what y refers to in this statement:

ALTER TABLE t ADD COLUMN a.b.c BIGINT AFTER y

It's meant to be a field within a.b, but that's not immediately obvious. It could, as well, be a top-level column and the statement could be a short hand for creating all the required nested fields with the root a after y.

The following structure would be more natural, but it only works with AFTER and it's incompatible with how nested columns are handled in general:

ALTER TABLE t ADD COLUMN c BIGINT AFTER a.b.y

All this is to say that I need to think more about the nested case.

[mosabua]

If we add FIRST .. why not also add LAST?

I guess that already exists.

[ebyhr]

Also .. is there any chance we implement this later for JDBC connectors and others across the board?

Yes, we can support this option in JDBC and other connectors if the datasource supports specifying columns positions.

[ebyhr]

Addressed comments.

[ebyhr]

FYI, putting fully qualified identifier after ADD COLUMN is same as Spark syntax. I don't stick to this approach, though.

[ebyhr]

We decided to exclude nested fields' support from this PR. I will file an issue and update this PR today or tomorrow.

[ebyhr]

(Rebased on master without any changes)