Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Column Exclusion #601

Merged
merged 18 commits into from
Nov 17, 2023
Merged

Column Exclusion #601

merged 18 commits into from
Nov 17, 2023

Conversation

serprex
Copy link
Contributor

@serprex serprex commented Nov 1, 2023

If a mirror is created with exclusions omit those columns from schema of target table

Syntax is in TABLE MAPPING of CREATE MIRROR, exclude:[col1,col2]

Ignores column appearing in schema changes too

Fixes #428

@serprex serprex requested a review from iskakaushik November 1, 2023 20:35
@saisrirampur saisrirampur marked this pull request as ready for review November 2, 2023 00:20
@iskakaushik iskakaushik marked this pull request as draft November 2, 2023 14:34
@serprex serprex force-pushed the exclude-columns branch 3 times, most recently from 80a25b1 to aa72c90 Compare November 3, 2023 01:33
flow/model/model.go Outdated Show resolved Hide resolved
flow/model/model.go Outdated Show resolved Hide resolved
@serprex serprex force-pushed the exclude-columns branch 5 times, most recently from f7a4072 to 834ea32 Compare November 10, 2023 22:39
flow/model/model.go Outdated Show resolved Hide resolved
@serprex serprex marked this pull request as ready for review November 10, 2023 23:08
@serprex serprex changed the title exclude columns Column Exclusion Nov 11, 2023
flow/workflows/setup_flow.go Outdated Show resolved Hide resolved
flow/connectors/postgres/postgres.go Show resolved Hide resolved
Copy link
Contributor

@iskakaushik iskakaushik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@serprex can we think about how we can test this change?

flow/activities/flowable.go Outdated Show resolved Hide resolved
@serprex
Copy link
Contributor Author

serprex commented Nov 13, 2023

Edgecase: with schema changes, drop excluded column. Expectation: ignore. Then add excluded column back. Expectation: continue excluding?

@iskakaushik
Copy link
Contributor

Edgecase: with schema changes, drop excluded column. Expectation: ignore. Then add excluded column back. Expectation: continue excluding?

Yes, continue excluding. Also currently mirrors are static, I think eventually we would want to make them dynamic, where a user can exclude more columns, but thats a problem for another day.

flow/model/model.go Outdated Show resolved Hide resolved
@serprex serprex force-pushed the exclude-columns branch 3 times, most recently from a052e93 to 94ab6f8 Compare November 16, 2023 23:11
@serprex serprex merged commit 0fe793e into main Nov 17, 2023
12 checks passed
iskakaushik pushed a commit that referenced this pull request Nov 17, 2023
If a mirror is created with exclusions omit those columns from schema of target table

Syntax is in `TABLE MAPPING` of `CREATE MIRROR`, `exclude:[col1,col2]`

Ignores column appearing in schema changes too

Fixes #428
@serprex serprex deleted the exclude-columns branch December 19, 2023 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support to exclude columns for tables
2 participants