feature/shopify-api-updates

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: Issue #60 and Issue #67

This PR will result in the following new package version: v0.9.0

This is not technically a breaking change, but due to the upstream changes in the source, this is considered breaking.

Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:

These changes are made breaking due to changes in the source.

• In June 2023 the Shopify connector received an update which upgraded the connector to be compatible with the new 2023-04 Shopify API. As a result, the following fields have been removed as they were deprecated in the API upgrade: (PR #70)

model	field removed
stg_shopify__customer	lifetime_duration
stg_shopify__order_line	fulfillment_service
stg_shopify__order_line	destination_location_* fields
stg_shopify__order_line	origin_location_* fields
stg_shopify__order	total_price_usd
stg_shopify__order	processing_method

• Please be aware that the removal of the fields from the staging models results in the removal of the fields in the relevant downstream models:

model	field removed
shopify__customer	lifetime_duration
shopify__customer_emails	lifetime_duration
shopify__order_lines	fulfillment_service
shopify__order_lines	destination_location_* fields
shopify__order_lines	origin_location_* fields
shopify__orders	total_price_usd
shopify__orders	processing_method

Documentation Updates

• The shopify_using_shop_metafields variable was added to the Adding Metafields of the README. It was erroneously omitted in a previous release.
• Documentation provided in the README for how to connect sources when leveraging the union schema/database feature.
• Removal of the current_total_price yml definition from the shopify__orders model as it was not being created in the model.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• dbt compile
• dbt run –full-refresh
• dbt run
• dbt test
• [n/a] dbt run –vars (if applicable)

Before marking this PR as "ready for review" the following have been applied:

• The appropriate issue has been linked and tagged
• You are assigned to the corresponding issue and this PR
• BuildKite integration tests are passing

Detailed Validation

Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":

• You have validated these changes and assure this PR will address the respective Issue/Feature.
• You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
• You have provided details below around the validation steps performed to gain confidence in these changes.

Not much validation was needed for these changes. I used the release notes as the basis for what fields were being deprecated. I then removed these fields from the staging models, macros, and yml documentation from the source package and then made similar adjustments to this package. I then ran the models and confirmed the fields were no longer present and that no errors resulted from the removal of the upstream fields in the downstream models. No further validation was necessary per my opinion.

Standard Updates

Please acknowledge that your PR contains the following standard updates:

• Package versioning has been appropriately indexed in the following locations:
  • indexed within dbt_project.yml
  • indexed within integration_tests/dbt_project.yml
• CHANGELOG has individual entries for each respective change in this PR
• README updates have been applied (if applicable)
• [n/a] DECISIONLOG updates have been updated (if applicable)
• Appropriate yml documentation has been added (if applicable)

dbt Docs

Please acknowledge that after the above were all completed the below were applied to your branch:

• docs were regenerated (unless this PR does not include any code or yml updates)

If you had to summarize this PR in an emoji, which would it be?

🛫

[fivetran-reneeli]

1. See notes in the shopify source PR

2. Also, should we remove variant_fulfillment_service from shopify__order_lines and shopify__inventory_levels , if that fulfillment_services field gets removed from the upstream product_variant model.

3. Finally, I'm noticing no models are showing up under files changed, for the ones with fields that are supposed to be removed. But I don't see the to-be removed fields in those models on this branch anyways... I must be missing something?
   

[fivetran-joemarkiewicz]

1. See notes in the shopify source PR

• See my responses from the source PR

2. Also, should we remove variant_fulfillment_service from shopify__order_lines and shopify__inventory_levels , if that fulfillment_services field gets removed from the upstream product_variant model.

• We are still keeping the fullfilment_service from the product_variant table as it is still able to be synced from the endpoint and is not listed as deprecated per the connector release notes.

3. Finally, I'm noticing no models are showing up under files changed, for the ones with fields that are supposed to be removed. But I don't see the to-be removed fields in those models on this branch anyways... I must be missing something?

• All the removed fields took place in the source package. They are removed from the end model because we are performing a select * in the respective models which now are returning less fields. As such, no models need to be changed but the ymls have been updated.

[fivetran-reneeli]

Thanks for explaining!

One last thing is I still don't see the shopify_using_shop_metafields bit in the readme

[fivetran-joemarkiewicz]

Thanks for explaining!

One last thing is I still don't see the shopify_using_shop_metafields bit in the readme

@fivetran-reneeli the metafield section is here

[fivetran-reneeli]

Got it! Ok, looks good to go