[Bug Fix] Remove incremental logic #97
Open
+251
−85
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Overview
This PR will address the following Issue/Feature: [#95]
This PR will result in the following new package version: v0.16.0
Source release is breaking, necessitating an upgrade.
Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:
Bug Fixes
shopify__discounts
shopify__order_lines
shopify__orders
shopify__transactions
merge
incremental strategy on BigQuery and Databricks, as we could not rely on a time series timestamp to impelment theinsert_overwrite
strategy. Usingmerge
is a costly strategy, so it defeats the purpose of leveraging incremental logic.new_vs_repeat
CTE logic inshopify__orders
was calculated within the specified incremental window but the new order was not in that same time period, it could be incorrectly processed as a new order.Upstream Under-the-Hood Updates from
shopify_source
Packageshopify_union_data
macro to accommodate Redshift's treatment of empty tables.null
row for Redshift destinations.varchar
. This throws errors in downstream transformations in theshopify
package. The 1 row will ensure that Redshift will respect the package's datatype casts.PR Checklist
Basic Validation
Please acknowledge that you have successfully performed the following commands locally:
Before marking this PR as "ready for review" the following have been applied:
Detailed Validation
Please share any and all of your validation steps:
If you had to summarize this PR in an emoji, which would it be?
🇪🇺