Add Support for Apache Iceberg on Snowflake

[aPeterHeise]

Snowflake added support for Apache Iceberg.

There seem to be two ways on how this will be implemented:

• external table with table_format specification (similar to Snowflake delta support #138), this will be an easy fix
• fully supported iceberg tables with full DML support, expected syntax: "create or replace iceberg table my_iceberg_table"

While this feature is still in private preview, I'd like to add it to the backlog here already.

[aPeterHeise]

Note, the way that #138 is implemented, option 1 via "external tables" should already work.

    - name: iceberg_tbl
         description: "External table using Iceberg files"
         external:
           location: "@stage"                      # reference an existing external stage
           file_format: "( type = parquet )"       # fully specified here, or reference an existing file format
           table_format: iceberg                   # specify the table format
           auto_refresh: false                     # requires configuring an event notification from Amazon S3 or Azure

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[ShahNewazKhan]

table_format: iceberg

table_format: iceberg is not supported for create or replace external table anymore.

[dataders]

@ShahNewazKhan are you suggesting that TABLE_FORMAT = ICEBERG is not possible in Snowflake's CREATE EXTERNAL TABLE command? therefore not possible to add support for it in dbt-external-tables?

[ShahNewazKhan]

@dataders correct, if you try to use CREATE EXTERNAL TABLE with TABLE_FORMAT = ICEBERG you get the following error:

You need to use CREATE OR REPLACE ICEBERG TABLE and specify an external_volume(similar to a snowflake named stage).

[afranzi]

For creating the external iceberg table we would need something like:

CREATE OR REPLACE ICEBERG TABLE {database}.{schema}.{table}
  EXTERNAL_VOLUME='{external_volume}'
  CATALOG='{glue_catalog_integration}'
  CATALOG_TABLE_NAME='{glue_table}'
  CATALOG_NAMESPACE='{glue_database}';

The existing code doesn't support it.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[dataders]

I'd really love to make this happen. If someone would be willing to pair with me it'd make the work much more pleasant and go faster!