Release/v0.16.0 (#149)

* bugfix/changing-transactions * release/v0.16.0 * updates * add integrity test * updates * update changelog & regen docs * update consistency test * update changelog * remove account_id updates and regen docs --------- Co-authored-by: Joe Markiewicz <[email protected]>
fivetran · Dec 2, 2024 · 379cb28 · 379cb28
1 parent 72f3fcf
commit 379cb28
Show file tree

Hide file tree

Showing 20 changed files with 97 additions and 71 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,16 @@
+# dbt_netsuite v0.16.0
+For Netsuite2, [PR #149](https://github.com/fivetran/dbt_netsuite/pull/149) includes the following updates: 
+
+## Breaking Changes (Full Refresh Required)
+- Revised the incremental logic of the `netsuite2__transaction_details` model to use `transaction_lines` CTE as the primary driver instead of `transactions`. 
+  - This ensures all transaction lines are captured, including those synced after the parent transaction.
+  - This also aligns with `transaction_lines` serving as the base CTE in the model, onto which all other CTEs are left-joined.
+  - When the `balance_sheet_transaction_detail_columns` and `income_statement_transaction_detail_columns` variables are used in the `netsuite2__balance_sheet` and `netsuite2__income_statement` models, all transactions are now included during incremental runs. This ensures no transactions are missed, aligning with the changes made in the `netsuite2__transaction_details` model.
+  - We still recommend running `dbt --full-refresh` periodically to maintain data quality of the models.
+
+## Documentation
+- Updated dbt documentation definitions.
+
 # dbt_netsuite v0.15.0
 For Netsuite2, [PR #144](https://github.com/fivetran/dbt_netsuite/pull/144) includes the following updates: 
 

diff --git a/README.md b/README.md
@@ -1,4 +1,6 @@
-<p align="center">
+# Netsuite Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_netsuite/))
+
+<p align="left">
     <a alt="License"
         href="https://github.com/fivetran/dbt_netsuite/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
@@ -13,7 +15,6 @@
         <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a>
 </p>
 
-# Netsuite Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_netsuite/))
 ## What does this dbt package do?
 - Produces modeled tables that leverage Netsuite data from [Fivetran's connector](https://fivetran.com/docs/applications/netsuite) in the format described by [this ERD](https://fivetran.com/docs/applications/netsuite#schemainformation) and builds off the output of our [Netsuite source package](https://github.com/fivetran/dbt_netsuite_source).
 - Enables users to insights into their netsuite data that can be used for financial statement reporting and deeper transactional analysis. This is achieved by the following:
@@ -101,7 +102,8 @@ Include the following netsuite package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/netsuite
-    version: [">=0.15.0", "<0.16.0"]
+    version: [">=0.16.0", "<0.17.0"]
+    
 ```
 ### Step 3: Define Netsuite.com or Netsuite2 Source
 As of April 2022 Fivetran made available a new Netsuite connector which leverages the Netsuite2 endpoint opposed to the original Netsuite.com endpoint. This package is designed to run for either or, not both. By default the `netsuite_data_model` variable for this package is set to the original `netsuite` value which runs the netsuite.com version of the package. If you would like to run the package on Netsuite2 data, you may adjust the `netsuite_data_model` variable to run the `netsuite2` version of the package.

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,6 +1,6 @@
 config-version: 2
 name: 'netsuite'
-version: '0.15.0'
+version: '0.16.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/integration_tests/.gitignore b/integration_tests/.gitignore
@@ -2,4 +2,5 @@ target/
 dbt_modules/
 dbt_packages/
 env/
-logs/
+logs/
+package-lock.yml
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'netsuite_integration_tests'
-version: '0.15.0'
+version: '0.16.0'
 profile: 'integration_tests'
 config-version: 2
 

diff --git a/integration_tests/seeds/netsuite2_accounting_period_data.csv b/integration_tests/seeds/netsuite2_accounting_period_data.csv
@@ -1,6 +1,6 @@
 id,_fivetran_deleted,_fivetran_synced,alllocked,allownonglchanges,aplocked,arlocked,closed,closedondate,date_deleted,enddate,isadjust,isinactive,isposting,isquarter,isyear,lastmodifieddate,parent,periodname,startdate
-5,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2002-03-31 0:00:00,,2002-03-31 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,2,Mar 2002,2002-03-01 00:00:00
-8,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2002-05-31 0:00:00,,2002-05-31 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,6,May 2002,2002-05-01 00:00:00
-9,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2002-06-30 0:00:00,,2002-06-30 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,6,Jun 2002,2002-06-01 00:00:00
+280,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2002-03-31 0:00:00,,2002-03-31 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,2,Mar 2002,2002-03-01 00:00:00
+330,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2002-05-31 0:00:00,,2002-05-31 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,6,May 2002,2002-05-01 00:00:00
+348,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2002-06-30 0:00:00,,2002-06-30 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,6,Jun 2002,2002-06-01 00:00:00
 21,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2001-02-28 0:00:00,,2001-02-28 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,19,Feb 2001,2001-02-01 00:00:00
 20,FALSE,2022-07-10 23:00:33,T,F,T,T,T,2001-01-31 0:00:00,,2001-01-31 0:00:00,F,F,T,F,F,2019-02-20 12:32:15,19,Jan 2001,2001-01-01 00:00:00
diff --git a/integration_tests/seeds/netsuite2_accounting_period_fiscal_cal_data.csv b/integration_tests/seeds/netsuite2_accounting_period_fiscal_cal_data.csv
@@ -1,7 +1,7 @@
 _fivetran_id,_fivetran_deleted,_fivetran_synced,accountingperiod,date_deleted,fiscalcalendar,parent
-t6+LTClexXfChh/pMhoj1F9SpSs=,FALSE,2022-07-10 22:17:10,245,,1,244
-hbwug8y6jPTTembEaYVZlsljZuQ=,FALSE,2022-07-10 22:17:10,252,,1,239
-hxkIkIIeDjsG8OgPAlvQVe9a6TE=,FALSE,2022-07-10 22:17:10,275,,1,274
+t6+LTClexXfChh/pMhoj1F9SpSs=,FALSE,2022-07-10 22:17:10,280,,1,244
+hbwug8y6jPTTembEaYVZlsljZuQ=,FALSE,2022-07-10 22:17:10,330,,1,239
+hxkIkIIeDjsG8OgPAlvQVe9a6TE=,FALSE,2022-07-10 22:17:10,348,,1,274
 H00Ert0t6kQHovZX4pHHC4N4d0Y=,FALSE,2022-07-10 22:17:10,6,,1,1
 KI+pra1uPLFPsBHFuvT/tXCaWdg=,FALSE,2022-07-10 22:17:10,184,,1,171
 JIpeyH5xsYzCF0YbPKr7obXrkhA=,FALSE,2022-07-10 22:17:10,218,,1,205
diff --git a/integration_tests/seeds/netsuite2_subsidiary_data.csv b/integration_tests/seeds/netsuite2_subsidiary_data.csv
@@ -1,3 +1,3 @@
 id,_fivetran_deleted,_fivetran_synced,country,currency,date_deleted,dropdownstate,edition,email,externalid,fax,federalidnumber,fiscalcalendar,fullname,iselimination,isinactive,languagelocale,lastmodifieddate,legalname,mainaddress,name,parent,purchaseorderamount,purchaseorderquantity,purchaseorderquantitydiff,receiptamount,receiptquantity,receiptquantitydiff,returnaddress,shippingaddress,showsubsidiaryname,ssnortin,state,state1taxnumber,tranprefix,url
-4,FALSE,2022-07-10 23:14:17,US,1,,AK,US,,,,,1,CeCe,F,F,en_US,2019-07-06 21:13:49,,1126,Meow Meow,3,,,,,,,,,F,,AK,,,
-6,FALSE,2022-05-27 13:56:54,US,1,,GA,US,,,,,1,Cat Jam,F,F,en_US,2017-06-13 5:39:53,,4659,Purr,3,,,,,,,,,F,,GA,,,
+4,FALSE,2022-07-10 23:14:17,US,1,,AK,US,,,,,1,CeCe,F,F,en_US,2019-07-06 21:13:49,,1126,Meow Meow,4,,,,,,,,,F,,AK,,,
+6,FALSE,2022-05-27 13:56:54,US,1,,GA,US,,,,,1,Cat Jam,F,F,en_US,2017-06-13 5:39:53,,4659,Purr,,,,,,,,,,F,,GA,,,
diff --git a/integration_tests/seeds/netsuite2_tran_acct_line_data.csv b/integration_tests/seeds/netsuite2_tran_acct_line_data.csv
@@ -1,6 +1,6 @@
 accountingbook,transaction,transactionline,_fivetran_deleted,_fivetran_synced,account,amount,amountpaid,amountunpaid,credit,date_deleted,debit,exchangerate,netamount,overheadparentitem,paymentamountunused,paymentamountused,posting,amountlinked
-1,4735,0,FALSE,2022-07-10 22:40:48,,,,,,,,1,,,,,T,0
-1,4455,0,FALSE,2022-07-10 22:40:48,,,,,,,,1,,,,,T,0
-1,2399,0,FALSE,2022-07-10 22:40:48,,,,,,,,1,,,,,T,0
+1,5914,5,FALSE,2022-07-10 22:40:48,133,,,,,,,1,,,,,T,0
+1,3,5,FALSE,2022-07-10 22:40:48,157,,,,,,,1,,,,,T,0
+1,302,4,FALSE,2022-07-10 22:40:48,118,,,,,,,1,,,,,T,0
 1,4458,0,FALSE,2022-07-10 22:40:48,,,,,,,,1,,,,,T,0
 1,4533,0,FALSE,2022-07-10 22:40:48,,,,,,,,1,,,,,T,0
diff --git a/integration_tests/seeds/netsuite2_transaction_data.csv b/integration_tests/seeds/netsuite2_transaction_data.csv
@@ -1,4 +1,4 @@
 _fivetran_synced,id,transactionnumber,type,memo,trandate,status,createddate,duedate,closedate,currency,entity,postingperiod,posting,isreversal,reversal,reversaldate,reversaldefer
 2022-05-27 13:41:13,5914,,VendBill,,2017-05-07 0:00:00,A,2017-05-07 10:34:43,2017-05-05 0:00:00,,1,1570,,T,F,325251,2023-08-01 0:00:00,F
 2022-05-27 13:41:13,3,,VendBill,,2015-09-10 0:00:00,B,2017-05-12 3:41:25,2015-10-09 0:00:00,2015-10-05 0:00:00,1,36,251,T,F,325252,2023-08-02 0:00:00,F
-2022-05-27 13:41:13,302,,VendBill,ZZZZZ,2016-03-20 0:00:00,B,2017-04-25 3:41:25,2016-04-19 0:00:00,2017-05-07 0:00:00,1,5,261,T,F,325323,2023-08-03 0:00:00,F
+2022-05-27 13:41:13,302,,VendBill,ZZZZZ,2016-03-20 0:00:00,B,2017-04-25 3:41:25,2016-04-19 0:00:00,2017-05-07 0:00:00,1,5,261,T,F,325323,2023-08-03 0:00:00,F
diff --git a/integration_tests/seeds/netsuite2_transaction_line_data.csv b/integration_tests/seeds/netsuite2_transaction_line_data.csv
@@ -3,4 +3,4 @@ id,transaction,_fivetran_deleted,_fivetran_synced,actualshipdate,amortizationend
 5,2187,FALSE,2022-05-27 13:29:17,,,,,,,T,,,,F,,,,,,F,,1,F,,,,,2186,,,,,,,,,,,,,,,,,,,,T,F,1296,,,,,2017-05-26 0:00:00,,,,,,,T,0,F,,F,F,F,F,F,F,F,F,252,,InvtPart,F,1,,F,2017-05-16 11:46:05,5,,F,F,The Witcher 3: The Wild Hunt,,F,,,,1,,-2,,0,,0,0,0,0,,,,,,,F,1,F,F,,,17717,,,F
 4,2248,FALSE,2022-05-27 13:29:17,,,,,,,T,,,,F,,,,,,F,,1,F,,,,,2247,,,,,,,,,,,,,,,,,,,,T,F,1049,,,,,,,,,,,,T,0,F,,F,F,F,F,F,F,F,F,385,WORK_ORDER,InvtPart,F,1,,F,2017-05-29 10:35:27,4,,F,F,Outer Wilds,,F,,,,1,,-4,,0,,0,0,0,0,,,,,,,F,1,F,F,,,29504,,,F
 3,725,FALSE,2022-05-27 13:29:17,,,,,,,F,,,,F,,,,,,F,,1,F,,,,,,,,,,,,,,,,,,,,,,,,,T,F,827,,,,,,,,,,,,T,0,F,,F,F,F,F,F,F,F,F,57,,InvtPart,T,2,,F,2017-05-02 10:13:14,3,,F,F,Xcom,,F,,,,-1,,-120,,0,,0,0,0,0,,,,,,,F,1,F,F,,,19043,,,F
-5,613,FALSE,2022-05-27 13:29:17,,,,,,,F,,,,F,,,,,,F,,1,F,,,,,,,,,,,,,,,,,,,,,,,,,T,F,737,,,,,,,,,,,,T,0,F,,F,F,F,F,F,F,F,F,53,,InvtPart,T,4,,F,2017-05-02 10:13:14,5,,F,F,Dark Souls,,F,,,,-1,,-10,,0,,0,0,0,0,,,,,,,F,1,F,F,,,27854,,,F
+5,613,FALSE,2022-05-27 13:29:17,,,,,,,F,,,,F,,,,,,F,,1,F,,,,,,,,,,,,,,,,,,,,,,,,,T,F,737,,,,,,,,,,,,T,0,F,,F,F,F,F,F,F,F,F,53,,InvtPart,T,4,,F,2017-05-02 10:13:14,5,,F,F,Dark Souls,,F,,,,-1,,-10,,0,,0,0,0,0,,,,,,,F,1,F,F,,,27854,,,F
diff --git a/integration_tests/tests/consistency/consistency_transaction_details.sql b/integration_tests/tests/consistency/consistency_transaction_details.sql
@@ -4,12 +4,12 @@
 ) }}
 
 with prod as (
-    select *
+    select {{ dbt_utils.star(from=ref('netsuite2__transaction_details'), except=var('netsuite_consistency_exclude_columns', '[]')) }}
     from {{ target.schema }}_netsuite_prod.netsuite2__transaction_details
 ),
 
 dev as (
-    select *
+    select {{ dbt_utils.star(from=ref('netsuite2__transaction_details'), except=var('netsuite_consistency_exclude_columns', '[]')) }}
     from {{ target.schema }}_netsuite_dev.netsuite2__transaction_details
 ),
 

diff --git a/integration_tests/tests/integrity/integrity_transaction_details.sql b/integration_tests/tests/integrity/integrity_transaction_details.sql
@@ -0,0 +1,30 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+{# This test is to check if the transaction_details has the same number of transactions 
+as the source transaction lines table after joining with the transactions source.
+This is important when making incremental logic changes. #}
+
+with stg_transaction_count as (
+    select count(distinct tl.transaction_id) as stg_count
+    from {{ target.schema }}_netsuite_dev.stg_netsuite2__transaction_lines tl
+    join {{ target.schema }}_netsuite_dev.stg_netsuite2__transactions tr
+        using(transaction_id)
+),
+
+transaction_details_count as (
+    select count(distinct transaction_id) as final_count
+    from {{ target.schema }}_netsuite_dev.netsuite2__transaction_details
+),
+
+final as (
+    select *
+    from stg_transaction_count
+    join transaction_details_count
+        on stg_count != final_count
+)
+
+select *
+from final
diff --git a/models/netsuite2.yml b/models/netsuite2.yml
@@ -361,8 +361,8 @@ models:
       tests:
           - unique
           - not_null
-    - name: _fivetran_synced_date
-      description: _fivetran_synced formatted as a date for incremental logic use.
+    - name: transaction_line_fivetran_synced_date
+      description: _fivetran_synced of the transaction_line formatted as a date for incremental logic use.
     - name: department_id
       description: "{{ doc('department_id') }}"
     - name: entity_id

diff --git a/models/netsuite2/netsuite2__balance_sheet.sql b/models/netsuite2/netsuite2__balance_sheet.sql
@@ -11,16 +11,12 @@
     )
 }}
 
-{% if is_incremental() %}
-{% set max_fivetran_synced_date = netsuite.netsuite_lookback(from_date='max(_fivetran_synced_date)', datepart='day', interval=var('lookback_window', 3)) %}
-{% endif %}
-
 with transactions_with_converted_amounts as (
     select * 
     from {{ref('int_netsuite2__tran_with_converted_amounts')}}
 
     {% if is_incremental() %}
-    where _fivetran_synced_date >= {{ max_fivetran_synced_date }}
+    where _fivetran_synced_date >= {{ netsuite.netsuite_lookback(from_date='max(_fivetran_synced_date)', datepart='day', interval=var('lookback_window', 3)) }}
     {% endif %}
 ), 
 
@@ -29,10 +25,6 @@ with transactions_with_converted_amounts as (
 transaction_details as (
     select * 
     from {{ ref('netsuite2__transaction_details') }}
-
-    {% if is_incremental() %}
-    where _fivetran_synced_date >= {{ max_fivetran_synced_date }}
-    {% endif %}
 ), 
 {% endif %}
 
@@ -265,7 +257,7 @@ balance_sheet as (
     left join transaction_details
         on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id
         and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id
-
+        
         {% if var('netsuite2__multibook_accounting_enabled', false) %}
         and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id
         {% endif %}

diff --git a/models/netsuite2/netsuite2__income_statement.sql b/models/netsuite2/netsuite2__income_statement.sql
@@ -11,16 +11,12 @@
     )
 }}
 
-{% if is_incremental() %}
-{% set max_fivetran_synced_date = netsuite.netsuite_lookback(from_date='max(_fivetran_synced_date)', datepart='day', interval=var('lookback_window', 3)) %}
-{% endif %}
-
 with transactions_with_converted_amounts as (
     select * 
     from {{ ref('int_netsuite2__tran_with_converted_amounts') }}
 
     {% if is_incremental() %}
-    where _fivetran_synced_date >= {{ max_fivetran_synced_date }}
+    where _fivetran_synced_date >= {{ netsuite.netsuite_lookback(from_date='max(_fivetran_synced_date)', datepart='day', interval=var('lookback_window', 3))  }}
     {% endif %}
 ), 
 
@@ -29,10 +25,6 @@ with transactions_with_converted_amounts as (
 transaction_details as (
     select * 
     from {{ ref('netsuite2__transaction_details') }}
-
-    {% if is_incremental() %}
-    where _fivetran_synced_date >= {{ max_fivetran_synced_date }}
-    {% endif %}
 ), 
 {% endif %}
 
@@ -172,6 +164,7 @@ income_statement as (
     join transaction_details
         on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id
         and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id
+
         {% if var('netsuite2__multibook_accounting_enabled', false) %}
         and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id
         {% endif %}

diff --git a/models/netsuite2/netsuite2__transaction_details.sql b/models/netsuite2/netsuite2__transaction_details.sql
@@ -2,28 +2,32 @@
     config(
         enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2'),
         materialized='table' if target.type in ('bigquery', 'databricks', 'spark') else 'incremental',
-        partition_by = {'field': '_fivetran_synced_date', 'data_type': 'date', 'granularity': 'month'}
-            if target.type not in ['spark', 'databricks'] else ['_fivetran_synced_date'],
+        partition_by = {'field': 'transaction_line_fivetran_synced_date', 'data_type': 'date', 'granularity': 'month'}
+            if target.type not in ['spark', 'databricks'] else ['transaction_line_fivetran_synced_date'],
         cluster_by = ['transaction_id'],
         unique_key='transaction_details_id',
         incremental_strategy = 'merge' if target.type in ('bigquery', 'databricks', 'spark') else 'delete+insert',
         file_format='delta'
     )
 }}
 
-{% if is_incremental() %}
-{% set max_fivetran_synced_date = netsuite.netsuite_lookback(from_date='max(_fivetran_synced_date)', datepart='day', interval=var('lookback_window', 3)) %}
-{% endif %}
+with transaction_lines as (
+    select 
+      *,
+      cast(_fivetran_synced as date) as transaction_line_fivetran_synced_date
 
-with transactions_with_converted_amounts as (
-    select * 
-    from {{ ref('int_netsuite2__tran_with_converted_amounts') }}
+    from {{ ref('int_netsuite2__transaction_lines') }}
 
     {% if is_incremental() %}
-    where _fivetran_synced_date >= {{ max_fivetran_synced_date }}
+    where cast(_fivetran_synced as date) >= {{ netsuite.netsuite_lookback(from_date='max(transaction_line_fivetran_synced_date)', datepart='day', interval=var('lookback_window', 3)) }}
     {% endif %}
 ),
 
+transactions_with_converted_amounts as (
+    select * 
+    from {{ ref('int_netsuite2__tran_with_converted_amounts') }}
+),
+
 accounts as (
     select * 
     from {{ ref('int_netsuite2__accounts') }}
@@ -39,18 +43,9 @@ subsidiaries as (
     from {{ var('netsuite2_subsidiaries') }}
 ),
 
-transaction_lines as (
-    select * 
-    from {{ ref('int_netsuite2__transaction_lines') }}
-),
-
 transactions as (
     select * 
     from {{ var('netsuite2_transactions') }}
-
-    {% if is_incremental() %}
-    where _fivetran_synced_date >= {{ max_fivetran_synced_date }}
-    {% endif %}
 ),
 
 customers as (
@@ -120,7 +115,7 @@ transaction_details as (
     transactions.transaction_date,
     transactions.due_date_at as transaction_due_date,
     transactions.transaction_type as transaction_type,
-    transactions._fivetran_synced_date,
+    transaction_lines.transaction_line_fivetran_synced_date,
     transactions.transaction_number,
     coalesce(transaction_lines.entity_id, transactions.entity_id) as entity_id,
     transactions.is_intercompany_adjustment as is_transaction_intercompany_adjustment,