Add support for v3 format in Iceberg #24455
Conversation
ebyhr
force-pushed
the
ebi/iceberg-v3-max
branch
2 times, most recently
from
62274e9 to
f48975a
Compare
|
I re-confirmed v3 spec. Looks like we should disallow the following write operation. Let me change the implementation. https://iceberg.apache.org/spec/#version-3
|
ebyhr
force-pushed
the
ebi/iceberg-v3-max
branch
from
f48975a to
2879820
Compare
|
Added a logic to disallow modifying rows with v3 to |
ebyhr
force-pushed
the
ebi/iceberg-v3-max
branch
2 times, most recently
from
f15c9d7 to
2b1f3ba
Compare
ebyhr
force-pushed
the
ebi/iceberg-v3-max
branch
from
2b1f3ba to
a798b1d
Compare
|
@raunaqmorarka Please take another look. |
ebyhr
force-pushed
the
ebi/iceberg-v3-max
branch
from
a798b1d to
2ced776
Compare
| @@ -865,8 +865,8 @@ connector using a {doc}`WITH </sql/create-table-as>` clause. | |||
| - Optionally specifies the file system location URI for the table. | |||
| * - `format_version` | |||
| - Optionally specifies the format version of the Iceberg specification to use | |||
| for new tables; either `1` or `2`. Defaults to `2`. Version `2` is required | |||
| for row level deletes. | |||
| for new tables; either `1`, `2` or `3`. Defaults to `2`. Only version `2` | |||
There was a problem hiding this comment.
I didn't get what exactly is the user facing impact of this change.
I see that we disallow altering table to v3 and update/merge.
Are reads, optimize and inserts to existing v3 tables allowed ? Is it valid to allow them while we don't have support for v3 deletion vectors ? Do we error out only if a v3 deletion vector is encountered ?
I think it would be nicer if we just added support for deletion vector along with v3 support, if not sure what's the benefit of supporting v3 without that.
There was a problem hiding this comment.
I will rework once they publish API for deletion vectors.
Description
This is preparatory tasks for adding support for v3 new features (e.g. variant type, timestamp nanos, deletion vectors and etc).
I didn't change the default version because the default value in Iceberg is still v2.
https://github.com/apache/iceberg/blob/fe2f593cd025223e4ab5ab41a296fb106ce3b1cf/core/src/main/java/org/apache/iceberg/TableMetadata.java#L54
Release notes