Union Data V2

CHANGELOG.md

@@ -1,3 +1,14 @@
+# dbt_zendesk v0.19.0
+[PR #178](https://github.com/fivetran/dbt_zendesk/pull/178) includes the following updates:
+
+## Feature Update: Run Models on Muliple Zendesk Sources
+- This release supports running the package on multiple Zendesk sources at once! See the [README](https://github.com/fivetran/dbt_zendesk?tab=readme-ov-file#step-3-define-database-and-schema-variables) for details on how to leverage this feature.
+
+> Please note: This is a **🚨Breaking Change🚨** in that we have a added a new field, `source_relation`, that points to the source connector from which the record originated. This field addition will require a `dbt run --full-refresh`, even if you are not using this new functionality.
+
+## Documentation
+- Cleaned up the column-level documentation descriptions for the `zendesk__ticket_enriched` and `zendesk__ticket_metrics` models.
+
 # dbt_zendesk v0.18.1
 [PR #174](https://github.com/fivetran/dbt_zendesk/pull/174) includes the following changes:
 

README.md

@@ -65,12 +65,13 @@ Include the following zendesk package version in your `packages.yml` file:
 ```yml
 packages:
   - package: fivetran/zendesk
-    version: [">=0.18.0", "<0.19.0"]
+    version: [">=0.19.0", "<0.20.0"]
 
 ```
 > **Note**: Do not include the Zendesk Support source package. The Zendesk Support transform package already has a dependency on the source in its own `packages.yml` file.
 
 ### Step 3: Define database and schema variables
+#### Option A: Single connector
 By default, this package runs using your destination and the `zendesk` schema. If this is not where your zendesk data is (for example, if your zendesk schema is named `zendesk_fivetran`), update the following variables in your root `dbt_project.yml` file accordingly:
 
 ```yml
@@ -79,6 +80,59 @@ vars:
     zendesk_schema: your_schema_name 
 ```
 
+#### Option B: Union multiple connectors
+If you have multiple Zendesk connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source each record came from in the `source_relation` column of each model. 
+
+To use this functionality, you will need to set the `zendesk_sources` variable in your root `dbt_project.yml` file:
+
+```yml
+# dbt_project.yml
+
+vars:
+  zendesk_sources:
+    - database: connector_1_destination_name # Required
+      schema: connector_1_schema_name # Rquired
+      name: connector_1_source_name # Required only if following the step in the following subsection
+
+    - database: connector_2_destination_name
+      schema: connector_2_schema_name
+      name: connector_2_source_name
+```
+
+##### Recommended: Incorporate unioned sources into DAG
+> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Zendesk connectors. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Zendesk source rather than one set of unioned models.*
+
+By default, this package defines one single-connector source, called `zendesk`, which will be disabled if you are unioning multiple connectors. This means that your DAG will not include your Zendesk sources, though the package will run successfully.
+
+To properly incorporate all of your Zendesk connectors into your project's DAG:
+1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_zendesk.yml` [file](https://github.com/fivetran/dbt_zendesk_source/blob/main/models/src_zendesk.yml#L15-L351).
+
+```yml
+# a .yml file in your root project
+sources:
+  - name: <name> # ex: Should match name in zendesk_sources
+    schema: <schema_name>
+    database: <database_name>
+    loader: fivetran
+    loaded_at_field: _fivetran_synced
+
+    freshness: # feel free to adjust to your liking
+      warn_after: {count: 72, period: hour}
+      error_after: {count: 168, period: hour}
+
+    tables: # copy and paste from zendesk_source/models/src_zendesk.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
+```
+
+> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_zendesk_source?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`. Otherwise, you may remove them from your source definition.
+
+2. Set the `has_defined_sources` variable (scoped to the `zendesk_source` package) to `True`, like such:
+```yml
+# dbt_project.yml
+vars:
+  zendesk_source:
+    has_defined_sources: true
+```
+
 ### Step 4: Enable/Disable models for non-existent sources
 This package takes into consideration that not every Zendesk Support account utilizes the `schedule`, `schedule_holiday`, `ticket_schedule`, `daylight_time`, `time_zone`, `audit_log`, `domain_name`, `user_tag`, `organization_tag`, or `ticket_form_history` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true`, except for `using_schedule_histories`. Add variables for only the tables you want to enable/disable:
 ```yml
@@ -230,7 +284,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
     - package: fivetran/zendesk_source
-      version: [">=0.13.0", "<0.14.0"]
+      version: [">=0.14.0", "<0.15.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'zendesk'
-version: '0.18.1'
+version: '0.19.0'
 
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]

integration_tests/dbt_project.yml

@@ -1,12 +1,16 @@
 config-version: 2
 
 name: 'zendesk_integration_tests'
-version: '0.18.1'
+version: '0.19.0'
 
 profile: 'integration_tests'
 
 vars:
   zendesk_schema: zendesk_integration_tests_50
+
+  using_organization_tags: false
+  using_holidays: false
+
   zendesk_source:
     zendesk_organization_identifier:           "organization_data"
     zendesk_schedule_identifier:               "schedule_data"
@@ -36,9 +40,10 @@ vars:
   # using_domain_names: false
   # using_user_tags: false
   # using_organization_tags: false
-  # fivetran_integrity_sla_metric_parity_exclusion_tickets: (56,80)
-  # fivetran_integrity_sla_first_reply_time_exclusion_tickets: (56,80)
-  # fivetran_consistency_sla_policies_exclusion_tickets: (55,58) # can remove after this release
+  fivetran_integrity_sla_metric_parity_exclusion_tickets: (56,80)
+  fivetran_integrity_sla_first_reply_time_exclusion_tickets: (56,80)
+  consistency_test_exclude_fields: ['source_relation', 'ticket_tags', 'ticket_day_id', 'assignee_ticket_last_update_at', 'assignee_last_login_at', 'requester_created_at', 'requester_updated_at', 'requester_ticket_last_update_at','requester_organization_created_at', 'requester_organization_updated_at', 'requester_last_login_at', 'created_at', 'updated_at'] 
+  # get rid of source_relation and ticket_day_id after my update ^
 
 models:
   +schema: "zendesk_{{ var('directed_schema','dev') }}"

integration_tests/tests/consistency/consistency_sla_policies.sql

@@ -76,4 +76,5 @@ select *
 from final
 where 
     {# Take differences in runtime into account #}
-    max_sla_elapsed_time - min_sla_elapsed_time > 2 
+    max_sla_elapsed_time - min_sla_elapsed_time > 5
+    and date(sla_applied_at) < current_date

integration_tests/tests/consistency/consistency_sla_policy_count.sql

@@ -9,7 +9,8 @@ with prod as (
         ticket_id,
         count(*) as total_slas
     from {{ target.schema }}_zendesk_prod.zendesk__sla_policies
-    {{ "where ticket_id not in " ~ var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) ~ "" if var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) }}
+    where date(sla_applied_at) < current_date
+    {{ "and ticket_id not in " ~ var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) ~ "" if var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) }}
     group by 1
 ),
 
@@ -18,7 +19,8 @@ dev as (
         ticket_id,
         count(*) as total_slas
     from {{ target.schema }}_zendesk_dev.zendesk__sla_policies
-    {{ "where ticket_id not in " ~ var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) ~ "" if var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) }}
+    where date(sla_applied_at) < current_date
+    {{ "and ticket_id not in " ~ var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) ~ "" if var('fivetran_consistency_sla_policy_count_exclusion_tickets',[]) }}
     group by 1
 ),
 

integration_tests/tests/consistency/consistency_ticket_backlog.sql

@@ -0,0 +1,51 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('zendesk__ticket_backlog'), except=var('consistency_test_exclude_fields', '[]')) }}
+    from {{ target.schema }}_zendesk_prod.zendesk__ticket_backlog
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('zendesk__ticket_backlog'), except=var('consistency_test_exclude_fields', '[]')) }}
+    from {{ target.schema }}_zendesk_dev.zendesk__ticket_backlog
+    {{ "where source_relation =  '" ~ (var("zendesk_database", target.database)|lower ~ "." ~ var("zendesk_schema", "zendesk")) ~ "'" if 'source_relation' in var("consistency_test_exclude_fields", '[]') }}
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final
+where date_day < current_date

integration_tests/tests/consistency/consistency_ticket_enriched.sql

@@ -0,0 +1,53 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('zendesk__ticket_enriched'), except=var('consistency_test_exclude_fields', '[]')) }}
+    from {{ target.schema }}_zendesk_prod.zendesk__ticket_enriched
+    where  true -- date(created_at) < current_date
+    and {{ dbt.datediff(dbt.current_timestamp(), "updated_at", "minute") }} >= 60
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('zendesk__ticket_enriched'), except=var('consistency_test_exclude_fields', '[]')) }}
+    from {{ target.schema }}_zendesk_dev.zendesk__ticket_enriched
+    where true -- date(created_at) < current_date
+    and {{ dbt.datediff(dbt.current_timestamp(), "updated_at", "minute") }} >= 60
+    {{ "and source_relation =  '" ~ (var("zendesk_database", target.database)|lower ~ "." ~ var("zendesk_schema", "zendesk")) ~ "'" if 'source_relation' in var("consistency_test_exclude_fields", '[]') }}
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_ticket_field_history.sql

@@ -0,0 +1,50 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('zendesk__ticket_field_history'), except=var('consistency_test_exclude_fields', '[]')) }}
+    from {{ target.schema }}_zendesk_prod.zendesk__ticket_field_history
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('zendesk__ticket_field_history'), except=var('consistency_test_exclude_fields', '[]')) }}
+    from {{ target.schema }}_zendesk_dev.zendesk__ticket_field_history
+    {{ "where source_relation =  '" ~ (var("zendesk_database", target.database)|lower ~ "." ~ var("zendesk_schema", "zendesk")) ~ "'" if 'source_relation' in var("consistency_test_exclude_fields", '[]') }}
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final
+where date_day < current_date

integration_tests/tests/consistency/consistency_ticket_metrics.sql

@@ -18,6 +18,7 @@ dev as (
         first_reply_time_business_minutes, 
         first_reply_time_calendar_minutes
     from {{ target.schema }}_zendesk_dev.zendesk__ticket_metrics
+    {{ "where source_relation =  '" ~ (var("zendesk_database", target.database)|lower ~ "." ~ var("zendesk_schema", "zendesk")) ~ "'" if 'source_relation' in var("consistency_test_exclude_fields", '[]') }}
 ),
 
 final as (