Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(iceberg): support iceberg connection #20189

Merged
merged 6 commits into from
Jan 18, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
support iceberg connection
  • Loading branch information
chenzl25 committed Jan 16, 2025
commit 39365e7f792336ab8986fc7cff6b83a05e1faebf
1 change: 1 addition & 0 deletions ci/scripts/e2e-iceberg-sink-v2-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ poetry run python main.py -t ./test_case/iceberg_source_position_delete.toml
poetry run python main.py -t ./test_case/iceberg_source_all_delete.toml
poetry run python main.py -t ./test_case/iceberg_source_explain_for_delete.toml
poetry run python main.py -t ./test_case/iceberg_predicate_pushdown.toml
poetry run python main.py -t ./test_case/iceberg_connection.toml

echo "--- Running benchmarks"
poetry run python main.py -t ./benches/predicate_pushdown.toml
Expand Down
67 changes: 67 additions & 0 deletions e2e_test/iceberg/test_case/iceberg_connection.slt
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
statement ok
set streaming_parallelism=4;

statement ok
CREATE TABLE s1 (i1 int, i2 varchar, i3 varchar);

statement ok
CREATE CONNECTION CONN WITH (
type = 'iceberg',
catalog.name = 'demo',
catalog.type = 'storage',
warehouse.path = 's3a://hummock001/iceberg-data',
s3.endpoint = 'http://127.0.0.1:9301',
s3.region = 'us-east-1',
s3.access.key = 'hummockadmin',
s3.secret.key = 'hummockadmin'
);

statement ok
CREATE SINK sink1 from s1 WITH (
connector = 'iceberg',
type = 'upsert',
database.name = 'demo_db',
table.name = 'test_connection_table',
connection = conn,
create_table_if_not_exists = 'true',
commit_checkpoint_interval = 1,
primary_key = 'i1,i2',
);

statement ok
INSERT INTO s1 (i1, i2, i3) values(1,'1','1'),(2,'2','2'),(3,'3','3'),(4,'4','4'),(5,'5','5');

statement ok
flush

statement ok
CREATE SOURCE iceberg_t1_source
WITH (
connector = 'iceberg',
connection = conn,
database.name = 'demo_db',
table.name = 'test_connection_table',
);

sleep 2s

query I
select * from iceberg_t1_source order by i1 limit 5;
----
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5

statement ok
DROP SINK sink1;

statement ok
DROP SOURCE iceberg_t1_source;

statement ok
DROP TABLE s1 cascade;

statement ok
DROP CONNECTION conn;
12 changes: 12 additions & 0 deletions e2e_test/iceberg/test_case/iceberg_connection.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
init_sqls = [
'CREATE SCHEMA IF NOT EXISTS demo_db',
'DROP TABLE IF EXISTS demo_db.test_connection_table',
]

slt = 'test_case/iceberg_connection.slt'

drop_sqls = [
'DROP TABLE IF EXISTS demo_db.test_connection_table',
'DROP SCHEMA IF EXISTS demo_db',
]
iceberg_connection
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There appears to be a stray iceberg_connection line at the end of this TOML file that should be removed. This line isn't valid TOML syntax and could interfere with proper file parsing.

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.

Loading