Update variable declarations (#155)

* Update variable declarations

* persist passthrough fixes

* Update variable configs + pass through metrics

* Update models + add docss

* PR fixes

* CHANGELOG

* PR fixes

* Update packages.yml

CHANGELOG.md

@@ -1,3 +1,13 @@
+# dbt_netsuite v0.17.1
+[PR #155](https://github.com/fivetran/dbt_netsuite/pull/155) includes the following updates: 
+
+## Macro Updates
+- Introduced a local version of the `persist_pass_through_columns` macro that directly calls the variables within our models. This removes the existing string-to-variable conversion and leads to cleaner parsing. 
+  - This new macro has no functional changes from the previous macro and will not require customers to make any changes on their end.
+- This new macro is applied to all end models with passthrough column functionality, and replaces the existing `persist_pass_through_columns` macro.
+- Models impacted for both `netsuite__*` and `netsuite2__*` include `balance_sheet`, `income_statement`, `transaction_details`.
+- The process for adding passthrough columns remains unchanged. [Consult the README](https://github.com/fivetran/dbt_netsuite?tab=readme-ov-file#optional-step-6-additional-configurations) for more details.
+
 # dbt_netsuite v0.17.0
 
 This release involves **breaking changes** and will require running a **full refresh**.
@@ -51,7 +61,6 @@ For Netsuite2, [PR #144](https://github.com/fivetran/dbt_netsuite/pull/144) incl
 
 
 > **IMPORTANT**: All of the affected models have pass-through functionality. If you have already been using passthrough column variables to include the newly added fields (without aliases), you **MUST** remove the fields from your passthrough variable configuration in order to avoid duplicate column errors.
-
 ## Feature Updates
 - You can now leverage passthrough columns in `netsuite2__transaction_details` to bring in additional fields from the `locations` and `subsidiaries` source tables. 
 - To add additional columns to this model, do so by adding our pass-through column variables `locations_pass_through_columns` and `subsidiaries_pass_through_columns` to your `dbt_project.yml` file:
@@ -64,21 +73,15 @@ vars:
         - name: "sub_field"
           alias: "subsidiary_field"
 ```
-
 - For more details on how to passthrough columns, [please consult our README section](https://github.com/fivetran/dbt_netsuite/blob/main/README.md#passing-through-additional-fields). 
-
 ## Under the Hood
 - Additional consistency tests added for each Netsuite2 end model in order to be used during integration test validations.
 - Updated yml documentation with new fields.
-
 ## Contributors
 - [@jmongerlyra](https://github.com/jmongerlyra) ([PR #136](https://github.com/fivetran/dbt_netsuite/pull/136))
 - [@fastbarreto](https://github.com/fastbarreto) ([PR #124](https://github.com/fivetran/dbt_netsuite/pull/124))
-
 # dbt_netsuite v0.14.0
-
 For Netsuite2, [PR #138](https://github.com/fivetran/dbt_netsuite/pull/138) and [PR #132](https://github.com/fivetran/dbt_netsuite/pull/132) include the following updates: 
-
 ## Breaking Changes (Full refresh required after upgrading)
 - Partitioned models have had the `partition_by` logic adjusted to include a granularity of a month. This change should only impact BigQuery warehouses and was applied to avoid the common `too many partitions` error users have experienced due to over-partitioning by day. Therefore, adjusting the partition to a monthly granularity will increase the partition windows and allow for more performant querying. 
 - This change was applied to the following models:

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'netsuite'
-version: '0.17.0'
+version: '0.17.1'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'netsuite_integration_tests'
-version: '0.17.0'
+version: '0.17.1'
 profile: 'integration_tests'
 config-version: 2
 
@@ -11,7 +11,7 @@ vars:
   netsuite_schema: netsuite_integration_tests_8
   netsuite_data_model_override: netsuite
 
-  # # Enable below when generating docs
+  # Enable below when generating docs
   # netsuite2__multibook_accounting_enabled: true
   # netsuite2__using_to_subsidiary: true
 

integration_tests/tests/consistency/consistency_balance_sheet.sql

@@ -9,7 +9,7 @@ with prod as (
 ),
 
 dev as (
-    select * except (transaction_amount, subsidiary_currency_symbol) -- remove after v0.17.0
+    select *
     from {{ target.schema }}_netsuite_dev.netsuite2__balance_sheet
 ),
 

integration_tests/tests/consistency/consistency_income_statement.sql

@@ -9,7 +9,7 @@ with prod as (
 ),
 
 dev as (
-    select * except (transaction_amount, subsidiary_currency_symbol) -- remove after v0.17.0
+    select *
     from {{ target.schema }}_netsuite_dev.netsuite2__income_statement
 ),
 

macros/persist_pass_through_columns.sql

@@ -0,0 +1,14 @@
+{% macro persist_pass_through_columns(pass_through_variable, identifier=none, transform='') %}
+
+{{ adapter.dispatch('persist_pass_through_columns', 'netsuite') (pass_through_variable, identifier=identifier, transform='') }}
+
+{%- endmacro %}
+
+
+{% macro default__persist_pass_through_columns(pass_through_variable, identifier=none, transform='') %}
+{% if pass_through_variable %}
+    {% for field in pass_through_variable %}
+    , {{ transform ~ '(' ~ (identifier ~ '.' if identifier else '') ~ (field.alias if field.alias else field.name) ~ ')' }} as {{ field.alias if field.alias else field.name }}
+    {% endfor %}
+{% endif %}
+{% endmacro %}

models/netsuite/netsuite__balance_sheet.sql

@@ -59,7 +59,7 @@ balance_sheet as (
         end as account_number
 
     --The below script allows for accounts table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
+    {{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', []), identifier='accounts') }},
 
     case
       when lower(accounts.is_balancesheet) = 'f' and lower(accounts.general_rate_type) in ('historical', 'average') then -converted_amount_using_transaction_accounting_period

models/netsuite/netsuite__income_statement.sql

@@ -67,19 +67,19 @@ income_statement as (
         subsidiaries.name as subsidiary_name
 
         --The below script allows for accounts table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
+        {{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', []), identifier='accounts') }},
 
         {{ dbt.concat(['accounts.account_number',"'-'", 'accounts.name']) }} as account_number_and_name,
         classes.full_name as class_full_name
 
         --The below script allows for classes table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('classes_pass_through_columns', identifier='classes') }},
+        {{ netsuite.persist_pass_through_columns(var('classes_pass_through_columns', []), identifier='classes') }},
 
         locations.full_name as location_full_name,
         departments.full_name as department_full_name
 
         --The below script allows for departments table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
+        {{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns', []), identifier='departments') }},
 
         -converted_amount_using_transaction_accounting_period as converted_amount,
         transactions_with_converted_amounts.account_category as account_category,

models/netsuite/netsuite__transaction_details.sql

@@ -93,10 +93,10 @@ transaction_details as (
     (lower(transactions.is_advanced_intercompany) = 'yes' or lower(transactions.is_intercompany) = 'yes') as is_transaction_intercompany
 
     --The below script allows for transactions table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('transactions_pass_through_columns', identifier='transactions') }}    
+    {{ netsuite.persist_pass_through_columns(var('transactions_pass_through_columns', []), identifier='transactions') }}    
 
     --The below script allows for transaction lines table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('transaction_lines_pass_through_columns', identifier='transaction_lines') }},
+    {{ netsuite.persist_pass_through_columns(var('transaction_lines_pass_through_columns', []), identifier='transaction_lines') }},
 
     accounting_periods.ending_at as accounting_period_ending,
     accounting_periods.full_name as accounting_period_full_name,
@@ -109,7 +109,7 @@ transaction_details as (
     accounts.account_number
 
     --The below script allows for accounts table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
+    {{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', []), identifier='accounts') }},
 
     lower(accounts.is_leftside) = 't' as is_account_leftside,
     lower(accounts.type_name) like 'accounts payable%' as is_accounts_payable,
@@ -140,7 +140,7 @@ transaction_details as (
     departments.name as department_name
 
     --The below script allows for departments table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
+    {{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns', []), identifier='departments') }},
 
     subsidiaries.name as subsidiary_name,
     case

models/netsuite2/netsuite2__balance_sheet.sql

@@ -124,7 +124,7 @@ balance_sheet as (
         else accounts.is_leftside
             end as is_account_leftside 
         --The below script allows for accounts table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
+        {{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', []), identifier='accounts') }},
 
         case
         when not accounts.is_balancesheet and lower(accounts.general_rate_type) in ('historical', 'average') then -converted_amount_using_transaction_accounting_period

models/netsuite2/netsuite2__income_statement.sql

@@ -102,22 +102,22 @@ income_statement as (
         subsidiaries_currencies.symbol as subsidiary_currency_symbol
 
         --The below script allows for accounts table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
+        {{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', []), identifier='accounts') }},
 
         {{ dbt.concat(['accounts.account_number',"'-'", 'accounts.name']) }} as account_number_and_name,
         classes.class_id,
         classes.full_name as class_full_name
 
         --The below script allows for accounts table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('classes_pass_through_columns', identifier='classes') }},
+        {{ netsuite.persist_pass_through_columns(var('classes_pass_through_columns', []), identifier='classes') }},
 
         locations.location_id,
         locations.full_name as location_full_name,
         departments.department_id,
         departments.full_name as department_full_name
 
         --The below script allows for departments table pass through columns.
-        {{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
+        {{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns', []), identifier='departments') }},
 
         transactions_with_converted_amounts.account_category as account_category,
         case when lower(accounts.account_type_id) = 'income' then 1

models/netsuite2/netsuite2__transaction_details.sql

@@ -125,10 +125,12 @@ transaction_details as (
     transactions.is_reversal_defer
 
     --The below script allows for transactions table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('transactions_pass_through_columns', identifier='transactions') }}
+
+    {{ netsuite.persist_pass_through_columns(var('transactions_pass_through_columns', []), identifier='transactions') }}
 
     --The below script allows for transaction lines table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('transaction_lines_pass_through_columns', identifier='transaction_lines') }},
+
+    {{ netsuite.persist_pass_through_columns(var('transaction_lines_pass_through_columns', []), identifier='transaction_lines') }},
 
     accounting_periods.ending_at as accounting_period_ending,
     accounting_periods.name as accounting_period_name,
@@ -143,7 +145,7 @@ transaction_details as (
     accounts.account_number
 
     --The below script allows for accounts table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
+    {{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', []), identifier='accounts') }},
 
     accounts.is_leftside as is_account_leftside,
     lower(accounts.account_type_id) = 'acctpay' as is_accounts_payable,
@@ -202,7 +204,7 @@ transaction_details as (
     locations.country as location_country
 
     -- The below script allows for locations table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('locations_pass_through_columns', identifier='locations') }},
+    {{ netsuite.persist_pass_through_columns(var('locations_pass_through_columns', []), identifier='locations') }},
 
     {% if var('netsuite2__using_vendor_categories', true) %}
     case 
@@ -239,15 +241,15 @@ transaction_details as (
     departments.name as department_name
 
     --The below script allows for departments table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
+    {{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns', []), identifier='departments') }},
 
     subsidiaries.subsidiary_id,
     subsidiaries.full_name as subsidiary_full_name,
     subsidiaries.name as subsidiary_name,
     subsidiaries_currencies.symbol as subsidiary_currency_symbol
 
     --The below script allows for subsidiaries table pass through columns.
-    {{ fivetran_utils.persist_pass_through_columns('subsidiaries_pass_through_columns', identifier='subsidiaries') }},
+    {{ netsuite.persist_pass_through_columns(var('subsidiaries_pass_through_columns', []), identifier='subsidiaries') }},
 
     case
       when lower(accounts.account_type_id) in ('income', 'othincome') then -transactions_with_converted_amounts.converted_amount_using_transaction_accounting_period