Merge pull request #45 from fivetran/release/v0.8.0

release/v0.8.0

.buildkite/scripts/run_models.sh

@@ -21,3 +21,5 @@ dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --vars '{using_address: false, using_bill: false, using_credit_memo: false, using_department: false, using_deposit: false, using_estimate: false, using_invoice: false, using_invoice_bundle: false, using_journal_entry: false, using_payment: false, using_refund_receipt: false, using_transfer: false, using_vendor_credit: false, using_sales_receipt: false}' --target "$db" --full-refresh
 dbt test --target "$db"
+
+dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -0,0 +1,51 @@
+## PR Overview
+**This PR will address the following Issue/Feature:**
+
+**This PR will result in the following new package version:**
+<!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
+
+**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+
+## PR Checklist
+### Basic Validation
+Please acknowledge that you have successfully performed the following commands locally:
+- [ ] dbt compile
+- [ ] dbt run –full-refresh
+- [ ] dbt run
+- [ ] dbt test
+- [ ] dbt run –vars (if applicable)
+
+Before marking this PR as "ready for review" the following have been applied:
+- [ ] The appropriate issue has been linked and tagged
+- [ ] You are assigned to the corresponding issue and this PR
+- [ ] BuildKite integration tests are passing
+
+### Detailed Validation
+Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
+- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
+- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
+- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+<!--- Provide the steps you took to validate your changes below. -->
+
+### Standard Updates
+Please acknowledge that your PR contains the following standard updates:
+- Package versioning has been appropriately indexed in the following locations:
+    - [ ] indexed within dbt_project.yml
+    - [ ] indexed within integration_tests/dbt_project.yml
+- [ ] CHANGELOG has individual entries for each respective change in this PR
+    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
+- [ ] README updates have been applied (if applicable)
+    <!--- Remember to check the following README locations for common updates. →
+        <!--- Suggested install range (needed for breaking changes) →
+        <!--- Dependency matrix is appropriately updated (if applicable) →
+        <!--- New variable documentation (if applicable) -->
+- [ ] DECISIONLOG updates have been updated (if applicable)
+- [ ] Appropriate yml documentation has been added (if applicable)
+
+### dbt Docs
+Please acknowledge that after the above were all completed the below were applied to your branch:
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
+
+### If you had to summarize this PR in an emoji, which would it be?
+<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
+:dancer:

.github/pull_request_template.md

@@ -1,52 +1,33 @@
-**Are you a current Fivetran customer?** 
-<!--- Please tell us your name, title and company -->
+**Please provide your name and company**
 
-**What change(s) does this PR introduce?** 
-<!--- Describe what changes your PR introduces to the package and how to leverage this new feature. -->
+**Link the issue/feature request which this PR is meant to address**
+<!--- If an issue was not created, please create one first so we may discuss the PR prior to opening one. -->
 
-**Did you update the CHANGELOG?** 
+**Detail what changes this PR introduces and how this addresses the issue/feature request linked above.**
+
+**How did you validate the changes introduced within this PR?**
+
+**Which warehouse did you use to develop these changes?**
+
+**Did you update the CHANGELOG?**
 <!--- Please update the new package version’s CHANGELOG entry detailing the changes included in this PR. -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Does this PR introduce a breaking change?**
-<!--- Does this PR introduce changes that will cause current package users' jobs to fail or require a `--full-refresh`? -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes (please provide breaking change details below.)
-- [ ] No  (please provide an explanation as to how the change is non-breaking below.)
-
-**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)** 
+**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)**
 <!--- The dbt_project.yml and the integration_tests/dbt_project.yml files contain the version number. Be sure to upgrade it accordingly -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Is this PR in response to a previously created Bug or Feature Request**
-<!--- If an Issue was created it is helpful to track the progress by linking it in the PR. -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes, Issue/Feature [link bug/feature number here]
-- [ ] No 
-
-**How did you test the PR changes?** 
-<!--- Proof of testing is required in order for the PR to be approved. -->
-<!--- To check a box, remove the space and insert an x in the box (eg. [x] BuildKite). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] BuildKite <!--- BuildKite testing is only applicable to Fivetran employees. --> 
-- [ ] Local (please provide additional testing details below)
-
-**Select which warehouse(s) were used to test the PR**
-<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] BigQuery
-- [ ] Redshift
-- [ ] Snowflake
-- [ ] Postgres
-- [ ] Databricks
-- [ ] Other (provide details below)
-
 **Provide an emoji that best describes your current mood**
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
 :dancer:
 
 **Feedback**
 
 We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next.
+
+**PR Template** 
+- [Community Pull Request Template](?expand=1&template=pull_request_template.md) (default)
+
+- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers)

CHANGELOG.md

@@ -1,3 +1,27 @@
+# dbt_quickbooks_source v0.8.0
+
+## 🎉 Feature Updates 🎉
+- Addition of the `detail_type` field within the `stg_quickbooks__invoice_line` model. This field will help identify the type of invoice line in the context of the entire invoice. ([PR #43](https://github.com/fivetran/dbt_quickbooks_source/pull/43))
+
+## Under the Hood
+- Casted the `department_id` within the following staging models to be of type string. This ensures downstream unions on this type are consistent and of the same type. ([PR #44](https://github.com/fivetran/dbt_quickbooks_source/pull/44))
+  - `stg_quickbooks__bill`
+  - `stg_quickbooks__bill_payment`
+  - `stg_quickbooks__credit_memo`
+  - `stg_quickbooks__department`
+  - `stg_quickbooks__deposit`
+  - `stg_quickbooks__estimate`
+  - `stg_quickbooks__invoice`
+  - `stg_quickbooks__journal_entry_line`
+  - `stg_quickbooks__purchase`
+  - `stg_quickbooks__refund_receipt`
+  - `stg_quickbooks__sales_receipt`
+  - `stg_quickbooks__vendor_credit`
+
+## Under the Hood:
+- Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job. ([#46](https://github.com/fivetran/dbt_quickbooks_source/pull/46))
+- Updated the pull request [templates](/.github). ([#46](https://github.com/fivetran/dbt_quickbooks_source/pull/46))
+
 # dbt_quickbooks_source v0.7.1
 ## 🎉 Feature Updates 🎉
 - Removed not-null test for `bill_payment_id` field in `stg_quickbooks__bill_linked_txn` to account for bills with empty bill payment values. ([#41](https://github.com/fivetran/dbt_quickbooks_source/pull/41))

README.md

@@ -52,7 +52,7 @@ Include the following QuickBooks package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/quickbooks_source
-    version: [">=0.7.0", "<0.8.0"]
+    version: [">=0.8.0", "<0.9.0"]
 ```
 
 ## Step 3: Define database and schema variables

dbt_project.yml

@@ -1,7 +1,8 @@
 config-version: 2
 name: 'quickbooks_source'
 
-version: '0.7.1' 
+version: '0.8.0' 
+
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

integration_tests/dbt_project.yml

@@ -1,5 +1,6 @@
 name: 'quickbooks_source_integration_tests'
-version: '0.7.1'
+version: '0.8.0'
+
 
 profile: 'integration_tests'
 config-version: 2

integration_tests/seeds/bill_data.csv

@@ -1,11 +1,11 @@
 id,_fivetran_deleted,_fivetran_synced,balance,created_at,currency_id,department_id,doc_number,due_date,exchange_rate,global_tax_calculation,home_balance,payable_account_id,private_note,sales_term_id,sync_token,total_amount,transaction_date,updated_at,vendor_id
-19,TRUE,2020-06-23 3:58:57,1069,2020-05-19 20:48:23,USD,,,,,,,,,,,588,,2020-05-20 12:36:09,
-17,TRUE,2020-06-23 3:58:57,992,2020-05-19 20:43:56,USD,,,,,,,,,,,974,,2020-05-22 16:39:03,
-15,TRUE,2020-06-23 3:58:57,800,2020-05-19 20:40:52,USD,,,,,,,,,,,386,,2020-06-05 20:14:19,
-14,TRUE,2020-06-23 3:58:57,572,2020-05-19 20:25:52,USD,,,,,,,,,,,666,,2020-06-05 20:14:12,
-16,TRUE,2020-06-23 3:58:57,737,2020-05-19 20:41:52,USD,,,,,,,,,,,878,,2020-05-19 20:41:55,
-18,TRUE,2020-06-23 3:58:57,213,2020-05-19 20:45:15,USD,,,,,,,,,,,946,,2020-05-22 16:40:28,
-398,FALSE,2020-12-17 19:32:00,52,2020-12-17 19:21:21,USD,,,2020-12-17,,,,37,,,0,822,2020-12-17,2020-12-17 19:21:21,20
-399,FALSE,2020-12-17 19:32:00,991,2020-12-17 19:28:35,USD,,2021-01-01,2020-12-17,,,,37,4e607e75b05d403ca078f850ccab9518,,0,647,2020-12-17,2020-12-17 19:28:35,6
-397,FALSE,2020-12-17 19:31:59,832,2020-12-17 19:21:02,USD,,,2020-12-17,,,,37,,,0,757,2020-12-17,2020-12-17 19:21:02,20
-401,FALSE,2020-12-17 19:32:00,454,2020-12-17 19:30:30,USD,,2021-03-01,2020-12-17,,,,37,6b681d0cbc9eeb22c903ccedfc47fe17,,0,485,2020-12-17,2020-12-17 19:30:30,6
+19,TRUE,2020-06-23 3:58:57,1069,2020-05-19 20:48:23,USD,1,,,,,,,,,,588,,2020-05-20 12:36:09,
+17,TRUE,2020-06-23 3:58:57,992,2020-05-19 20:43:56,USD,1,,,,,,,,,,974,,2020-05-22 16:39:03,
+15,TRUE,2020-06-23 3:58:57,800,2020-05-19 20:40:52,USD,1,,,,,,,,,,386,,2020-06-05 20:14:19,
+14,TRUE,2020-06-23 3:58:57,572,2020-05-19 20:25:52,USD,1,,,,,,,,,,666,,2020-06-05 20:14:12,
+16,TRUE,2020-06-23 3:58:57,737,2020-05-19 20:41:52,USD,1,,,,,,,,,,878,,2020-05-19 20:41:55,
+18,TRUE,2020-06-23 3:58:57,213,2020-05-19 20:45:15,USD,1,,,,,,,,,,946,,2020-05-22 16:40:28,
+398,FALSE,2020-12-17 19:32:00,52,2020-12-17 19:21:21,USD,1,,2020-12-17,,,,37,,,0,822,2020-12-17,2020-12-17 19:21:21,20
+399,FALSE,2020-12-17 19:32:00,991,2020-12-17 19:28:35,USD,1,2021-01-01,2020-12-17,,,,37,4e607e75b05d403ca078f850ccab9518,,0,647,2020-12-17,2020-12-17 19:28:35,6
+397,FALSE,2020-12-17 19:31:59,832,2020-12-17 19:21:02,USD,1,,2020-12-17,,,,37,,,0,757,2020-12-17,2020-12-17 19:21:02,20
+401,FALSE,2020-12-17 19:32:00,454,2020-12-17 19:30:30,USD,1,2021-03-01,2020-12-17,,,,37,6b681d0cbc9eeb22c903ccedfc47fe17,,0,485,2020-12-17,2020-12-17 19:30:30,6

integration_tests/seeds/invoice_line_data.csv

@@ -1,11 +1,11 @@
-index,invoice_id,_fivetran_synced,amount,bundle_id,bundle_quantity,description,description_service_date,description_tax_code_id,discount_account_id,discount_class_id,discount_discount_percent,discount_percent_based,discount_tax_code_id,sales_item_account_id,sales_item_class_id,sales_item_discount_amount,sales_item_discount_rate,sales_item_item_id,sales_item_quantity,sales_item_service_date,sales_item_tax_code_id,sales_item_unit_price,sub_total_item_id,item_id,quantity,tax_code_id,account_id
-0,35647,2021-01-18 6:16:26,27381.64,211,0.6993497,,,,,,,,,,,,,,,,,,,,,,
-5,35647,2021-01-18 6:16:26,24850,,,,,,,,,,,,,,,,,,,,,,,,
-2,35647,2021-01-18 6:16:26,0,,,0f9d69329f489637a6ae046f4091e365,,,,,,,,,,,,18,,,NON,,,,,,
-4,35647,2021-01-18 6:16:26,0,,,553bf7b55f32b6f7074ae45d987530ce,,,,,,,,,,,,18,,,NON,,,,,,
-3,35647,2021-01-18 6:16:26,-4280.01,,,2ae778d8d42424d33b656517b2c892bf,,,,,,,,224,,,,34,0.7,,NON,-6120,,,,,
-1,35647,2021-01-18 6:16:26,1748.37,,,,,,,,,,,211,,,,227,0.7,,NON,2500,,,,,
-0,40092,2021-01-19 2:18:38,21437.5,212,0.5,,,,,,,,,,,,,,,,,,,,,,
-10,40092,2021-01-19 2:18:38,17181.75,,,,,,,,,,,,,,,,,,,,,,,,
-4,40092,2021-01-19 2:18:38,0,,,7f3c88bf068c8edd47367bd68c01cd8f,,,,,,,,,,,,18,,,NON,,,,,,
-7,40092,2021-01-19 2:18:38,0,,,703d01ddc458e55f5b43da8c88aaa581,,,,,,,,,,,,18,,,NON,,,,,,
+index,invoice_id,_fivetran_synced,amount,bundle_id,bundle_quantity,description,description_service_date,description_tax_code_id,discount_account_id,discount_class_id,discount_discount_percent,discount_percent_based,discount_tax_code_id,sales_item_account_id,sales_item_class_id,sales_item_discount_amount,sales_item_discount_rate,sales_item_item_id,sales_item_quantity,sales_item_service_date,sales_item_tax_code_id,sales_item_unit_price,sub_total_item_id,item_id,quantity,tax_code_id,account_id,detail_type
+0,35647,2021-01-18 6:16:26,27381.64,211,0.6993497,,,,,,,,,,,,,,,,,,,,,,,SalesItemLineDetail
+5,35647,2021-01-18 6:16:26,24850,,,,,,226,,,,,,,,,,,,,,,,,,,DiscountLineDetail
+2,35647,2021-01-18 6:16:26,0,,,0f9d69329f489637a6ae046f4091e365,,,,,,,,,,,,18,,,NON,,,,,,,SalesItemLineDetail
+4,35647,2021-01-18 6:16:26,0,,,553bf7b55f32b6f7074ae45d987530ce,,,,,,,,,,,,18,,,NON,,,,,,,
+3,35647,2021-01-18 6:16:26,-4280.01,,,2ae778d8d42424d33b656517b2c892bf,,,,,,,,224,,,,34,0.7,,NON,-6120,,,,,,
+1,35647,2021-01-18 6:16:26,1748.37,,,,,,,,,,,,,,,,,,NON,2500,,,,,,DescriptionOnly
+0,40092,2021-01-19 2:18:38,21437.5,212,0.5,,,,,,,,,,,,,,,,,,,,,,,SalesItemLineDetail
+10,40092,2021-01-19 2:18:38,17181.75,,,,,,,,,,,,,,,,,,,,,,,,,SubTotalLineDetail
+4,40092,2021-01-19 2:18:38,0,,,7f3c88bf068c8edd47367bd68c01cd8f,,,,,,,,,,,,,,,NON,,,,,,,SubTotalLineDetail
+7,40092,2021-01-19 2:18:38,0,,,703d01ddc458e55f5b43da8c88aaa581,,,,,,,,,,,,,,,NON,,,,,,,SubTotalLineDetail

integration_tests/seeds/purchase_data.csv

@@ -1,11 +1,11 @@
 id,_fivetran_deleted,_fivetran_synced,account_id,created_at,credit,currency_id,customer_id,department_id,doc_number,employee_id,exchange_rate,global_tax_calculation,payment_method_id,payment_type,print_status,private_note,remit_to_address_id,sync_token,tax_code_id,total_amount,total_tax,transaction_date,transaction_source,updated_at,vendor_id
-162,true,2020-12-30 15:02:20,,2020-11-06 1:19:33,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:19:33,18
-236,true,2020-12-30 15:02:20,,2020-11-06 1:24:25,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:24:25,18
-146,true,2020-12-30 15:02:20,,2020-11-06 1:18:27,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:18:27,18
-64,true,2020-09-01 21:02:33,,2020-07-15 20:03:18,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-07-15 20:03:18,18
-194,true,2020-12-30 15:02:20,,2020-11-06 1:21:34,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:21:34,18
-103,true,2020-12-30 15:02:20,,2020-09-25 3:42:57,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-10-15 14:17:12,18
-206,true,2020-12-30 15:02:20,,2020-11-06 1:22:25,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:22:25,18
-150,true,2020-12-30 15:02:20,,2020-11-06 1:18:45,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:18:45,18
-266,true,2020-12-30 15:02:20,,2020-11-21 3:21:38,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-12-02 20:19:49,
-260,true,2020-12-30 15:02:20,,2020-11-07 1:43:36,,USD,,,,,,,,CreditCard,,,,,,,,,,2020-11-07 1:43:36,18
+107,FALSE,2020-12-30 15:02:20,,2020-11-06 1:19:33,,USD,,1,,,,,,CreditCard,,,,,,,,2020-11-06 1:19:33,,2020-11-06 1:19:33,18
+101,FALSE,2020-12-30 15:02:20,,2020-11-06 1:24:25,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:24:25,18
+103,FALSE,2020-12-30 15:02:20,,2020-11-06 1:18:27,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:18:27,18
+232,FALSE,2020-09-01 21:02:33,,2020-07-15 20:03:18,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-07-15 20:03:18,18
+228,FALSE,2020-12-30 15:02:20,,2020-11-06 1:21:34,,USD,,2,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:21:34,18
+372,FALSE,2020-12-30 15:02:20,,2020-09-25 3:42:57,,USD,,2,,,,,,CreditCard,,,,,,,,,,2020-10-15 14:17:12,18
+190,FALSE,2020-12-30 15:02:20,,2020-11-06 1:22:25,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:22:25,18
+340,FALSE,2020-12-30 15:02:20,,2020-11-06 1:18:45,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-11-06 1:18:45,18
+342,FALSE,2020-12-30 15:02:20,,2020-11-21 3:21:38,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-12-02 20:19:49,
+350,FALSE,2020-12-30 15:02:20,,2020-11-07 1:43:36,,USD,,1,,,,,,CreditCard,,,,,,,,,,2020-11-07 1:43:36,18

macros/get_invoice_line_columns.sql

@@ -14,6 +14,7 @@
     {"name": "discount_discount_percent", "datatype": dbt.type_float()},
     {"name": "discount_percent_based", "datatype": "boolean"},
     {"name": "discount_tax_code_id", "datatype": dbt.type_string()},
+    {"name": "detail_type", "datatype": dbt.type_string()},
     {"name": "index", "datatype": dbt.type_string()},
     {"name": "invoice_id", "datatype": dbt.type_string()},
     {"name": "item_id", "datatype": dbt.type_string()},

models/src_quickbooks.yml

@@ -490,6 +490,8 @@ sources:
             description: "{{ doc('item_id') }} associated with the invoice line."
           - name: quantity
             description: Quantity of the item included on the invoice line.
+          - name: detail_type
+            description: Individual line items of a transaction. Valid line types include SalesItemLineDetail, DescriptionOnly (also used for inline Subtotal lines), DiscountLineDetail and SubTotalLineDetail (used for the overall transaction). FYI - Records synced prior to 11/2021 will have a null value.
 
       - name: invoice_line_bundle
         identifier: "{{ var('quickbooks_invoice_line_bundle_identifier', 'invoice_line_bundle') }}"

models/stg_quickbooks.yml

@@ -582,6 +582,8 @@ models:
         description: Reference to the item associated with the invoice line.
       - name: quantity
         description: Quantity of the item included on the invoice line.
+      - name: detail_type
+        description: Individual line items of a transaction. Valid line types include SalesItemLineDetail, DescriptionOnly (also used for inline Subtotal lines), DiscountLineDetail and SubTotalLineDetail (used for the overall transaction). FYI - Records synced prior to 11/2021 will have a null value.
 
   - name: stg_quickbooks__invoice_line_bundle
     description: "{{ doc('invoice_line_bundle_table') }}"

models/stg_quickbooks__bill.sql

@@ -42,7 +42,7 @@ final as (
         balance,
         cast(doc_number as {{ dbt.type_string() }}) as doc_number,
         currency_id,
-        department_id,
+        cast(department_id as {{ dbt.type_string() }}) as department_id,
         due_date as due_date_at,
         transaction_date,
         exchange_rate,

models/stg_quickbooks__bill_payment.sql

@@ -44,7 +44,7 @@ final as (
         cast(credit_card_account_id as {{ dbt.type_string() }}) as credit_card_account_id,
         exchange_rate,
         currency_id,
-        department_id,
+        cast(department_id as {{ dbt.type_string() }}) as department_id,
         pay_type,
         total_amount,
         transaction_date,