added tests for PG and SF schema changes, fixed edge cases and quoting #480

heavycrystal · 2023-10-04T17:08:46Z

This PR adds on to the existing work towards syncing schema changes in CDC mirrors (#368):

handles an edge case where a column is dropped and recreated with a different data type in the same ALTER TABLE statement.
Adds quoting, guarding against case-sensitive column names and reserved identifiers.
Fixes type mapping for BigQuery

Additionally, adds some tests for ReplayTableSchemaDelta for Postgres and Snowflake. Bigquery does not have tests yet but has been manually tested.

flow/connectors/postgres/postgres_schema_delta_test.go

flow/e2e/snowflake/snowflake_schema_delta_test.go

flow/connectors/snowflake/snowflake.go

flow/connectors/postgres/cdc.go

flow/connectors/postgres/postgres.go

added tests for PG and SF schema changes, fixed edge cases and quoting

03a8f0a

heavycrystal requested a review from iskakaushik October 4, 2023 17:08

github-actions bot reviewed Oct 4, 2023

View reviewed changes

heavycrystal and others added 5 commits October 4, 2023 23:03

fixed lints pt.1

85db75b

reverting previous sfTypeToQValueKind map

90954c5

fix schema mapping for snowflake to float64

c456fd7

Merge branch 'main' into schema-changes-tests

44fa3fb

Merge branch 'main' into schema-changes-tests

571a343

iskakaushik approved these changes Oct 5, 2023

View reviewed changes

iskakaushik merged commit 5a2a4aa into main Oct 5, 2023
17 of 19 checks passed

heavycrystal mentioned this pull request Oct 16, 2023

Mirrored Table schema changes #421

Closed

serprex deleted the schema-changes-tests branch July 19, 2024 15:17

Provide feedback