add datatype casting and temp deps (#12)

* add datatype casting and temp deps

* run on empty schema

* working with limit 1

* stage directory stuff for showing people

* workgin

* testing

* docs

* polish

* changelog

* update source package ref

---------

Co-authored-by: Jamie Rodriguez <[email protected]>

CHANGELOG.md

@@ -1,3 +1,17 @@
+# dbt_qualtrics v0.3.0
+
+[PR #12](https://github.com/fivetran/dbt_qualtrics/pull/12) includes the following updates:
+
+## Under the Hood
+- Switched from using `dbt.current_timestamp_backcompat()` to the more up-to-date `dbt.current_timestamp_backcompat()` [macro](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros#current_timestamp) in the `qualtrics__daily_breakdown` model.
+- (Maintainers only) Adds consistency and integrity (row count) tests for each end model.
+
+### [Upstream Under-the-Hood Updates](https://github.com/fivetran/dbt_qualtrics_source/blob/main/CHANGELOG.md) from `qualtrics_source` Package
+- Explicitly casts all boolean fields as `{{ dbt.type_boolean() }}`.
+- (Affects Redshift only) Creates new `qualtrics_union_data` macro to accommdate Redshift's treatment of empty tables.
+  - For each staging model, if the source table is not found in any of your schemas, the package will create a empty table with 0 rows for non-Redshift warehouses and a table with 1 all-`null` row for Redshift destinations.
+  - This is necessary as Redshift will ignore explicit data casts when a table is completely empty and materialize every column as a `varchar`. This throws errors in dowstream transformations in the `zendesk` package. The 1 row will ensure that Redshift will respect the package's datatype casts.
+
 # dbt_qualtrics v0.2.0
 
 [PR #8](https://github.com/fivetran/dbt_qualtrics/pull/8) includes the following updates: 

README.md

@@ -57,7 +57,7 @@ Include the following qualtrics package version in your `packages.yml` file:
 ```yml
 packages:
   - package: fivetran/qualtrics
-    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 Do **NOT** include the `qualtrics_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
@@ -139,7 +139,7 @@ models:
 ```
 
 #### Change the source table references
-If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable. This config is available only when running the package on a single connector.
 
 > IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_qualtrics_source/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
@@ -166,7 +166,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
     - package: fivetran/qualtrics_source
-      version: [">=0.2.0", "<0.3.0"]
+      version: [">=0.3.0", "<0.4.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'qualtrics'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"] 
 

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'qualtrics_integration_tests'
-version: '0.2.0'
+version: '0.3.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -12,6 +12,9 @@ dispatch:
   - macro_namespace: dbt_utils
     search_order: ['spark_utils', 'dbt_utils']
 
+models:
+  +schema: "qualtrics_{{ var('directed_schema','dev') }}"
+
 seeds:
   qualtrics_integration_tests:
     +column_types:

integration_tests/packages.yml

@@ -1,2 +1,2 @@
 packages:
-  - local: ../
+  - local: ../

integration_tests/tests/consistency/consistency_qualtrics__contact.sql

@@ -0,0 +1,51 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__contact'), except=["avg_survey_duration_in_seconds"]) }},
+        round(avg_survey_duration_in_seconds, 2) as avg_survey_duration_in_seconds
+    from {{ target.schema }}_qualtrics_prod.qualtrics__contact
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__contact'), except=["avg_survey_duration_in_seconds"]) }},
+        round(avg_survey_duration_in_seconds, 2) as avg_survey_duration_in_seconds
+    from {{ target.schema }}_qualtrics_dev.qualtrics__contact
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_qualtrics__daily_breakdown.sql

@@ -0,0 +1,49 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__daily_breakdown')) }}
+    from {{ target.schema }}_qualtrics_prod.qualtrics__daily_breakdown
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__daily_breakdown')) }}
+    from {{ target.schema }}_qualtrics_dev.qualtrics__daily_breakdown
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_qualtrics__directory.sql

@@ -0,0 +1,49 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__directory')) }}
+    from {{ target.schema }}_qualtrics_prod.qualtrics__directory
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__directory')) }}
+    from {{ target.schema }}_qualtrics_dev.qualtrics__directory
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_qualtrics__distribution.sql

@@ -0,0 +1,49 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__distribution')) }}
+    from {{ target.schema }}_qualtrics_prod.qualtrics__distribution
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__distribution')) }}
+    from {{ target.schema }}_qualtrics_dev.qualtrics__distribution
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_qualtrics__response.sql

@@ -0,0 +1,49 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__response')) }}
+    from {{ target.schema }}_qualtrics_prod.qualtrics__response
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__response')) }}
+    from {{ target.schema }}_qualtrics_dev.qualtrics__response
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_qualtrics__survey.sql

@@ -0,0 +1,53 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__survey'), except=['avg_response_duration_in_seconds', 'avg_survey_progress_pct']) }},
+        round(avg_response_duration_in_seconds, 0) as avg_response_duration_in_seconds,
+        round(avg_survey_progress_pct, 0) as avg_survey_progress_pct
+    from {{ target.schema }}_qualtrics_prod.qualtrics__survey
+),
+
+dev as (
+    select
+        {{ dbt_utils.star(from=ref('qualtrics__survey'), except=['avg_response_duration_in_seconds', 'avg_survey_progress_pct']) }},
+        round(avg_response_duration_in_seconds, 0) as avg_response_duration_in_seconds,
+        round(avg_survey_progress_pct, 0) as avg_survey_progress_pct
+    from {{ target.schema }}_qualtrics_dev.qualtrics__survey
+
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final