Merge pull request #42 from fivetran/feature/dbt-utils-star

[Feature] Replacing select * with dbt_utils.star in our tmp models

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -4,48 +4,22 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present)
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
-<!--- Provide the steps you took to validate your changes below. -->
-
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
-### If you had to summarize this PR in an emoji, which would it be?
-<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+Please share any and all of your validation steps:
+<!--- Provide the steps you took to validate your changes below. -->

CHANGELOG.md

@@ -1,3 +1,15 @@
+# dbt_zendesk_source v0.10.0
+[PR #42](https://github.com/fivetran/dbt_zendesk_source/pull/42) introduces the following updates: 
+
+# 🚨 Breaking Change (Snowflake users) 🚨
+- We have changed the identifier logic in `src_zendesk.yml` to account for `group` being both a Snowflake reserved word and a source table. Snowflake users will want to execute a `dbt run --full-refresh` before using the new version of the package.
+
+# 🎉 Feature Update 🎉 
+- Updated our `tmp` models to utilize the `dbt_utils.star` macro rather than the select * function. This removes Snowflake issues that arise when a source's dimensions change. 
+
+## 🔎 Under the Hood 🔎 
+- Updates to the seed files and seed file configurations for the package integration tests to ensure updates are properly tested.
+
 # dbt_zendesk_source v0.9.0
 
 # 🚨 New Schedule Holiday Table 🚨

README.md

@@ -92,6 +92,16 @@ If an individual source table has a different name than the package expects, add
 vars:
     zendesk_<default_source_table_name>_identifier: your_table_name 
 ```
+
+### 🚨 Snowflake Users
+If you do **not** use the default all-caps naming conventions for Snowflake, you may need to provide the case-sensitive spelling of your source tables that are also Snowflake reserved words. 
+
+In this package, this would apply to the `GROUP` source. If you are receiving errors for this source, include the below identifier in your `dbt_project.yml` file:
+
+```yml
+vars:
+    zendesk_group_identifier: "Group" # as an example, must include the double-quotes and correct case!
+```  
 
 </details>
 

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'zendesk_source'
-version: '0.9.0'
+version: '0.10.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   zendesk_source:

integration_tests/.gitignore

@@ -3,4 +3,6 @@ target/
 dbt_modules/
 dbt_packages
 logs/
-.DS_Store
+.DS_Store
+env/
+package-lock.yml

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: zendesk_source_integrations_tests_2
+      schema: zendesk_source_integrations_tests_5
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: zendesk_source_integrations_tests_2
+      schema: zendesk_source_integrations_tests_5
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: zendesk_source_integrations_tests_2
+      schema: zendesk_source_integrations_tests_5
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: zendesk_source_integrations_tests_2
+      schema: zendesk_source_integrations_tests_5
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: zendesk_source_integrations_tests_2
+      schema: zendesk_source_integrations_tests_5
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,12 +1,12 @@
 config-version: 2
 
 name: 'zendesk_source_integration_tests'
-version: '0.9.0'
+version: '0.10.0'
 
 profile: 'integration_tests'
 
 vars:
-  zendesk_schema: zendesk_source_integrations_tests_2
+  zendesk_schema: zendesk_source_integrations_tests_5
   zendesk_source:
     zendesk_organization_identifier:           "organization_data"
     zendesk_schedule_identifier:               "schedule_data"
@@ -20,11 +20,12 @@ vars:
     zendesk_domain_name_identifier:            "domain_name_data"
     zendesk_time_zone_identifier:              "time_zone_data"
     zendesk_daylight_time_identifier:          "daylight_time_data"
-    group:                                     "{{ ref('group_data') }}" 
-    brand:                                     "{{ fivetran_utils.seed_data_helper('brand_data',['postgres']) }}" 
-    user_tag:                                  "{{ fivetran_utils.snowflake_seed_data('user_tag_data') }}" 
-    organization_tag:                          "{{ fivetran_utils.snowflake_seed_data('organization_tag_data') }}" 
-    user:                                      "{{ fivetran_utils.snowflake_seed_data('user_data') }}" 
+    zendesk_brand_identifier:                  "brand_data"
+    zendesk_group_identifier:                  "group_data" 
+    zendesk_organization_tag_identifier:       "organization_tag_data"
+    zendesk_user_identifier:                   "user_data"
+    zendesk_user_tag_identifier:               "user_tag_data"
+
 
     using_schedules:            True
     using_domain_names:         True
@@ -69,6 +70,7 @@ seeds:
         id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         created_at: timestamp
         due_at: timestamp
+        updated_at: timestamp
         assignee_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         brand_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         external_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
@@ -99,20 +101,28 @@ seeds:
         last_login_at: timestamp
         updated_at: timestamp
       +enabled: "{{ true if target.type != 'snowflake' else false }}"
-    user_tag_data:
-      +column_types:
-        user_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
-        _fivetran_synced: timestamp
-    brand_data_postgres:
+    user_data_snowflake:
+      +alias: user_data
       +column_types:
         id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
-        _fivetran_synced: timestamp
-      +enabled: "{{ true if target.type == 'postgres' else false }}"
+        external_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        locale_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        organization_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        created_at: timestamp
+        last_login_at: timestamp
+        updated_at: timestamp
+      +enabled: "{{ true if target.type == 'snowflake' else false }}"
     brand_data:
       +column_types:
         id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         _fivetran_synced: timestamp
       +enabled: "{{ true if target.type != 'postgres' else false }}"
+    brand_data_postgres:
+      +alias: brand_data
+      +column_types:
+        id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        _fivetran_synced: timestamp
+      +enabled: "{{ true if target.type == 'postgres' else false }}"
     ticket_form_history_data:
       +column_types:
         id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
@@ -121,21 +131,21 @@ seeds:
       +column_types:
         organization_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         _fivetran_synced: timestamp
-    organization_tag_data:
+    user_tag_data:
       +column_types:
-        organization_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        user_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         _fivetran_synced: timestamp
-    user_data_snowflake:
-      +column_types:
-        id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
-        external_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
-        locale_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
-        organization_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
-        created_at: timestamp
-      +enabled: "{{ true if target.type == 'snowflake' else false }}"
+      +enabled: "{{ true if target.type != 'snowflake' else false }}"  
     user_tag_data_snowflake:
+      +alias: user_tag_data
       +enabled: "{{ true if target.type == 'snowflake' else false }}"
+    organization_tag_data:
+      +column_types:
+        organization_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        _fivetran_synced: timestamp
+      +enabled: "{{ true if target.type != 'snowflake' else false }}"  
     organization_tag_data_snowflake:
+      +alias: organization_tag_data
       +enabled: "{{ true if target.type == 'snowflake' else false }}"
 
 dispatch:

integration_tests/requirements.txt

@@ -4,4 +4,4 @@ dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.3.0,<2.0.0
+dbt-databricks>=1.6.0,<2.0.0

integration_tests/seeds/ticket_data.csv

@@ -1,5 +1,5 @@
 id,_fivetran_synced,allow_channelback,assignee_id,brand_id,created_at,description,due_at,external_id,forum_topic_id,group_id,has_incidents,is_public,organization_id,priority,problem_id,recipient,requester_id,status,subject,submitter_id,system_client,ticket_form_id,type,updated_at,url,via_channel,via_source_from_id,via_source_from_title,via_source_rel,via_source_to_address,via_source_to_name,merged_ticket_ids,via_source_from_address,followup_ids,via_followup_source_id
-1595,2020-03-20 02:32:49.426,FALSE,,360003529474,2020-02-19 01:54:52,description1,,,,360006965034,FALSE,TRUE,370295712714,,,[email protected],396331237134,deleted,subject1,396331237134,,360002048693,,2020-02-19 01:55:11,https://zendesk.com/api/v2/tickets/1595.json,web,,,,[email protected],,[],,,
+1595,2020-03-20 02:32:49.426,FALSE,,360003529474,2020-02-19 01:54:52,description1,,,,360006965034,FALSE,TRUE,370295712714,,,[email protected],396331237134,deleted,subject1,396331237134,,360002048693,incident,2020-02-19 01:55:11,https://zendesk.com/api/v2/tickets/1595.json,web,,,,[email protected],,[],,,
 16988,2021-01-13 20:09:16.325,FALSE,418284131934,360003529474,2020-12-22 00:19:23,description1,,,,360013366274,FALSE,TRUE,370469077513,,,[email protected],1500656884401,solved,subject1,1500656884401,,360002048693,,2021-01-13 18:42:39,https://zendesk.com/api/v2/tickets/16988.json,email,,,,[email protected],Support,[],,[],
 14173,2020-11-11 20:08:45.130,FALSE,396371699653,360003529474,2020-10-28 12:03:02,description1,,,,360006965034,FALSE,TRUE,370321120273,,,[email protected],424883466453,closed,subject1,424883466453,,360002048693,,2020-11-11 17:01:32,https://zendesk.com/api/v2/tickets/14173.json,email,,,,[email protected],Support,[],,,
 11071,2020-10-02 14:08:33.216,FALSE,,360003529474,2020-08-28 18:06:36,description1,,,,,FALSE,TRUE,,,,[email protected],419755385214,deleted,subject1,419755385214,,360002048693,,2020-09-02 11:01:27,https://zendesk.com/api/v2/tickets/11071.json,email,,,,X,Support,[],,,