Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: SQL targets should allow configurable insert vs upsert behavior by default #1783

Closed
pnadolny13 opened this issue Jun 21, 2023 · 1 comment · Fixed by #1893
Closed
Assignees
Labels
kind/Feature New feature or request valuestream/SDK

Comments

@pnadolny13
Copy link
Contributor

Similar to #1781 we should add a default configuration enum to SQL targets to configure whether to run the target in insert (append only) mode or upsert mode. Some targets will want this behavior so they'll implement it in a custom way, its better to get in front of it so we have a consistent config option and enum for all targets.

In addition we should consider recommending insert as the default option even if upsert is support and a key property is provided. This avoids challenges with deduplicating within a sync (i.e. unsorted streams, no timestamp or incrementing sort key, etc.).

@pnadolny13
Copy link
Contributor Author

There could also be the option to run in trunc and load mode. This mode would truncate the table prior to loading. Its a bit risky so we should document it well. Snowflake has the time travel feature that allows you to recover any dropped tables or truncated rows back to a certain timestamp (default 24hr retention) so its a little safer if the developer accidentally uses this feature and drops their data.

@tayloramurphy tayloramurphy changed the title SQL targets should allow configurable insert vs upsert behavior by default feat: SQL targets should allow configurable insert vs upsert behavior by default Jul 11, 2023
@pnadolny13 pnadolny13 self-assigned this Aug 3, 2023
@github-project-automation github-project-automation bot moved this to Planned in Data Team Sep 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Feature New feature or request valuestream/SDK
Projects
Status: Planned
Development

Successfully merging a pull request may close this issue.

2 participants