Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update variable declarations #155

Merged
merged 8 commits into from
Jan 15, 2025
+90 −246
Merged

Update variable declarations #155

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
11688c6
Update variable declarations
fivetran-avinash Jan 3, 2025
36b4c69
persist passthrough fixes
fivetran-avinash Jan 3, 2025
712eee7
Update variable configs + pass through metrics
fivetran-avinash Jan 6, 2025
dd8ca1c
Update models + add docss
fivetran-avinash Jan 6, 2025
3a6f652
PR fixes
fivetran-avinash Jan 10, 2025
02c226b
CHANGELOG
fivetran-avinash Jan 10, 2025
a863ba0
PR fixes
fivetran-avinash Jan 13, 2025
50da8d6
Update packages.yml
fivetran-avinash Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
# dbt_netsuite v0.17.1
[PR #155](https://github.com/fivetran/dbt_netsuite/pull/155) includes the following updates:

## Macro Updates
- Introduced a local version of the `persist_pass_through_columns` macro that directly calls the variables within our models. This removes the existing string-to-variable conversion and leads to cleaner parsing.
fivetran-joemarkiewicz marked this conversation as resolved.
Show resolved Hide resolved
- This new macro is applied to all end models with passthrough column functionality, and replaces the existing `persist_pass_through_columns` macro.
- Models impacted for both `netsuite__*` and `netsuite2__*` include `balance_sheet`, `income_statement`, `transaction_details`.
- The process for adding passthrough columns remains unchanged. [Consult the README](https://github.com/fivetran/dbt_netsuite?tab=readme-ov-file#optional-step-6-additional-configurations) for more details.

# dbt_netsuite v0.17.0

This release involves **breaking changes** and will require running a **full refresh**.
Expand Down Expand Up @@ -64,7 +73,6 @@ vars:
- name: "sub_field"
alias: "subsidiary_field"
```

- For more details on how to passthrough columns, [please consult our README section](https://github.com/fivetran/dbt_netsuite/blob/main/README.md#passing-through-additional-fields).

## Under the Hood
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
config-version: 2
name: 'netsuite'
version: '0.17.0'
version: '0.17.1'
require-dbt-version: [">=1.3.0", "<2.0.0"]

models:
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json
View file
Open in desktop

Large diffs are not rendered by default.

274 changes: 63 additions & 211 deletions docs/index.html
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions integration_tests/dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'netsuite_integration_tests'
version: '0.17.0'
version: '0.17.1'
profile: 'integration_tests'
config-version: 2

Expand All @@ -11,9 +11,9 @@ vars:
netsuite_schema: netsuite_integration_tests_8
netsuite_data_model_override: netsuite

# # Enable below when generating docs
# netsuite2__multibook_accounting_enabled: true
# netsuite2__using_to_subsidiary: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obligatory reminder: Does this need to be uncommented before merge?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this only needs to enabled when doing the docs generation. After that, it should be commented back out.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh i meant re-commented*

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh d'oh. Yes, should be recommented.

# Enable below when generating docs
netsuite2__multibook_accounting_enabled: true
netsuite2__using_to_subsidiary: true

netsuite_source:
# Netsuite Seed Data
Expand Down
2 changes: 1 addition & 1 deletion integration_tests/tests/consistency/consistency_balance_sheet.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ with prod as (
),

dev as (
select * except (transaction_amount, subsidiary_currency_symbol) -- remove after v0.17.0
select *
from {{ target.schema }}_netsuite_dev.netsuite2__balance_sheet
),

Expand Down
2 changes: 1 addition & 1 deletion integration_tests/tests/consistency/consistency_income_statement.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ with prod as (
),

dev as (
select * except (transaction_amount, subsidiary_currency_symbol) -- remove after v0.17.0
select *
from {{ target.schema }}_netsuite_dev.netsuite2__income_statement
),

Expand Down
14 changes: 14 additions & 0 deletions macros/persist_pass_through_columns.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{% macro persist_pass_through_columns(pass_through_variable, identifier=none, transform='') %}

{{ adapter.dispatch('persist_pass_through_columns', 'netsuite') (pass_through_variable, identifier=identifier, transform='') }}

{%- endmacro %}


{% macro default__persist_pass_through_columns(pass_through_variable, identifier=none, transform='') %}
{% if pass_through_variable %}
{% for field in pass_through_variable %}
, {{ transform ~ '(' ~ (identifier ~ '.' if identifier else '') ~ (field.alias if field.alias else field.name) ~ ')' }} as {{ field.alias if field.alias else field.name }}
{% endfor %}
{% endif %}
{% endmacro %}
2 changes: 1 addition & 1 deletion models/netsuite/netsuite__balance_sheet.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ balance_sheet as (
end as account_number

--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
{{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', none), identifier='accounts') }},

case
when lower(accounts.is_balancesheet) = 'f' and lower(accounts.general_rate_type) in ('historical', 'average') then -converted_amount_using_transaction_accounting_period
Expand Down
6 changes: 3 additions & 3 deletions models/netsuite/netsuite__income_statement.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,19 +67,19 @@ income_statement as (
subsidiaries.name as subsidiary_name

--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
{{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', none), identifier='accounts') }},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we use None here and not [] like we usually do?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Especially since that's how we have it defined in the dbt_project.yml

Copy link
Contributor Author

@fivetran-avinash fivetran-avinash Jan 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I was following what was defined in our persist passthrough macro code and assumed the none condition was what we wanted. Updated this in all macro calls.


{{ dbt.concat(['accounts.account_number',"'-'", 'accounts.name']) }} as account_number_and_name,
classes.full_name as class_full_name

--The below script allows for classes table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('classes_pass_through_columns', identifier='classes') }},
{{ netsuite.persist_pass_through_columns(var('classes_pass_through_columns', none), identifier='classes') }},

locations.full_name as location_full_name,
departments.full_name as department_full_name

--The below script allows for departments table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
{{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns', none), identifier='departments') }},

-converted_amount_using_transaction_accounting_period as converted_amount,
transactions_with_converted_amounts.account_category as account_category,
Expand Down
8 changes: 4 additions & 4 deletions models/netsuite/netsuite__transaction_details.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,10 +93,10 @@ transaction_details as (
(lower(transactions.is_advanced_intercompany) = 'yes' or lower(transactions.is_intercompany) = 'yes') as is_transaction_intercompany

--The below script allows for transactions table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('transactions_pass_through_columns', identifier='transactions') }}
{{ netsuite.persist_pass_through_columns(var('transactions_pass_through_columns', none), identifier='transactions') }}

--The below script allows for transaction lines table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('transaction_lines_pass_through_columns', identifier='transaction_lines') }},
{{ netsuite.persist_pass_through_columns(var('transaction_lines_pass_through_columns', none), identifier='transaction_lines') }},

accounting_periods.ending_at as accounting_period_ending,
accounting_periods.full_name as accounting_period_full_name,
Expand All @@ -109,7 +109,7 @@ transaction_details as (
accounts.account_number

--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
{{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns', none), identifier='accounts') }},

lower(accounts.is_leftside) = 't' as is_account_leftside,
lower(accounts.type_name) like 'accounts payable%' as is_accounts_payable,
Expand Down Expand Up @@ -140,7 +140,7 @@ transaction_details as (
departments.name as department_name

--The below script allows for departments table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
{{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns', none), identifier='departments') }},

subsidiaries.name as subsidiary_name,
case
Expand Down
2 changes: 1 addition & 1 deletion models/netsuite2/netsuite2__balance_sheet.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ balance_sheet as (
else accounts.is_leftside
end as is_account_leftside
--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
{{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns'), identifier='accounts') }},

case
when not accounts.is_balancesheet and lower(accounts.general_rate_type) in ('historical', 'average') then -converted_amount_using_transaction_accounting_period
Expand Down
6 changes: 3 additions & 3 deletions models/netsuite2/netsuite2__income_statement.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,22 +102,22 @@ income_statement as (
subsidiaries_currencies.symbol as subsidiary_currency_symbol

--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
{{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns'), identifier='accounts') }},

{{ dbt.concat(['accounts.account_number',"'-'", 'accounts.name']) }} as account_number_and_name,
classes.class_id,
classes.full_name as class_full_name

--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('classes_pass_through_columns', identifier='classes') }},
{{ netsuite.persist_pass_through_columns(var('classes_pass_through_columns'), identifier='classes') }},

locations.location_id,
locations.full_name as location_full_name,
departments.department_id,
departments.full_name as department_full_name

--The below script allows for departments table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
{{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns'), identifier='departments') }},

transactions_with_converted_amounts.account_category as account_category,
case when lower(accounts.account_type_id) = 'income' then 1
Expand Down
14 changes: 8 additions & 6 deletions models/netsuite2/netsuite2__transaction_details.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,12 @@ transaction_details as (
transactions.is_reversal_defer

--The below script allows for transactions table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('transactions_pass_through_columns', identifier='transactions') }}

{{ netsuite.persist_pass_through_columns(var('transactions_pass_through_columns'), identifier='transactions') }}

--The below script allows for transaction lines table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('transaction_lines_pass_through_columns', identifier='transaction_lines') }},

{{ netsuite.persist_pass_through_columns(var('transaction_lines_pass_through_columns'), identifier='transaction_lines') }},

accounting_periods.ending_at as accounting_period_ending,
accounting_periods.name as accounting_period_name,
Expand All @@ -143,7 +145,7 @@ transaction_details as (
accounts.account_number

--The below script allows for accounts table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('accounts_pass_through_columns', identifier='accounts') }},
{{ netsuite.persist_pass_through_columns(var('accounts_pass_through_columns'), identifier='accounts') }},

accounts.is_leftside as is_account_leftside,
lower(accounts.account_type_id) = 'acctpay' as is_accounts_payable,
Expand Down Expand Up @@ -202,7 +204,7 @@ transaction_details as (
locations.country as location_country

-- The below script allows for locations table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('locations_pass_through_columns', identifier='locations') }},
{{ netsuite.persist_pass_through_columns(var('locations_pass_through_columns'), identifier='locations') }},

{% if var('netsuite2__using_vendor_categories', true) %}
case
Expand Down Expand Up @@ -239,15 +241,15 @@ transaction_details as (
departments.name as department_name

--The below script allows for departments table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('departments_pass_through_columns', identifier='departments') }},
{{ netsuite.persist_pass_through_columns(var('departments_pass_through_columns'), identifier='departments') }},

subsidiaries.subsidiary_id,
subsidiaries.full_name as subsidiary_full_name,
subsidiaries.name as subsidiary_name,
subsidiaries_currencies.symbol as subsidiary_currency_symbol

--The below script allows for subsidiaries table pass through columns.
{{ fivetran_utils.persist_pass_through_columns('subsidiaries_pass_through_columns', identifier='subsidiaries') }},
{{ netsuite.persist_pass_through_columns(var('subsidiaries_pass_through_columns'), identifier='subsidiaries') }},

case
when lower(accounts.account_type_id) in ('income', 'othincome') then -transactions_with_converted_amounts.converted_amount_using_transaction_accounting_period
Expand Down
5 changes: 3 additions & 2 deletions packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
packages:
- package: fivetran/netsuite_source
version: [">=0.11.0", "<0.12.0"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obligatory reminder to update this before release

- git: https://github.com/fivetran/dbt_netsuite_source.git
revision: update/netsuite-variable-declarations
warn-unpinned: false
fivetran-avinash marked this conversation as resolved.
Show resolved Hide resolved