Feature/add action result

CHANGELOG.md

@@ -1,5 +1,23 @@
-# dbt_marketo v0.11.0
+# dbt_marketo v0.12.0
+[PR #39](https://github.com/fivetran/dbt_marketo/pull/39) includes the following updates:
+
+## Breaking Changes:
+- The `action_result` field is now included in the following models, allowing users to filter records based on the `action_result` value.
+    - `stg_marketo__activity_send_email`
+    - `marketo__email_sends_deduped`
+    - `marketo__email_sends`
+  - *Note:* If you have previously added this field via the `marketo__activity_send_email_passthrough_columns` variable, remove or alias it there to prevent duplicate column errors.
+- To minimize customer impact, we limited adding `action_result` to models where it does not alter the grain. Users needing `action_result` in aggregate models can derive it from the `marketo__email_sends` model using custom aggregation logic. For more details, refer to the [DECISIONLOG entry](https://github.com/fivetran/dbt_marketo/blob/main/CHANGELOG.md#action-result-field-introduction).
+
+## Documentation updates:
+- Added missing definitions to dbt documentation.
 
+## Under the Hood:
+- Added consistency integration tests for the models listed above.
+- Removed all tests from intermediate models as they were unnecessary and to optimize resource usage.
+- Updated seed data to include `action_result`.
+
+# dbt_marketo v0.11.0
 [PR #33](https://github.com/fivetran/dbt_marketo/pull/33) includes the following updates:
 ## Bug Fix
 - Removed the use of `ignore nulls` statements in `marketo__lead_history` and `marketo__change_data_scd`, which was incompatible with PostgreSQL and Databricks Runtime. The logic has been updated with a new approach but produces the same results as before.

DECISIONLOG.md

@@ -0,0 +1,12 @@
+## Decision Log
+
+### `action_result` Field Introduction
+We added the `action_result` field to capture email action outcomes in the following tables:
+
+- `stg_marketo__activity_send_email`
+- `marketo__email_sends_deduped`
+- `marketo__email_sends`
+
+Initially, we considered adding `action_result` to further downstream models, such as `marketo__email_stats__by_email_template`, `marketo__email_stats__by_lead`, `marketo__email_templates`, and `marketo__leads`. However, this would require a finer grain, potentially doubling the size of these tables.
+
+To minimize customer impact, we limited adding `action_result` to models where it does not alter data grain. Users needing `action_result` in aggregate models can derive it from the `marketo__email_sends` model using custom aggregation logic.

README.md

@@ -59,7 +59,7 @@ Include the following Marketo package version in your `packages.yml` file.
 ```yml
 packages:
   - package: fivetran/marketo
-    version: [">=0.11.0", "<0.12.0"]
+    version: [">=0.12.0", "<0.13.0"]
 ```
 Do **NOT** include the `marketo_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 
@@ -155,7 +155,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
     - package: fivetran/marketo_source
-      version: [">=0.11.0", "<0.12.0"]
+      version: [">=0.12.0", "<0.13.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'marketo'
-version: '0.11.0'
+version: '0.12.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 on-run-start: "{{ lead_history_columns_warning() }}"

integration_tests/dbt_project.yml

@@ -1,9 +1,13 @@
 name: 'marketo_integration_tests'
-version: '0.11.0'
+version: '0.12.0'
 profile: 'integration_tests'
 config-version: 2
 
 vars:
+    # enable when regenerating docs
+    marketo__enable_campaigns: true
+    marketo__enable_programs: true
+
     marketo_schema: marketo_integration_tests_6
     marketo_source:
         marketo_lead_identifier: "marketo_lead_data"

integration_tests/tests/consistency/consistency_email_sends.sql

@@ -0,0 +1,49 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select {{ dbt_utils.star(
+        from=ref('marketo__email_sends'), 
+        except=['activity_id'] + var('consistency_test_exclude_metrics', [])) }} -- marketo__email_sends_deduped does not create a consistent activity_id
+    from {{ target.schema }}_marketo_prod.marketo__email_sends
+),
+
+dev as (
+    select {{ dbt_utils.star(
+        from=ref('marketo__email_sends'),
+        except=['activity_id'] + var('consistency_test_exclude_metrics', [])) }}
+    from {{ target.schema }}_marketo_dev.marketo__email_sends
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_email_templates.sql

@@ -0,0 +1,45 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select {{ dbt_utils.star(from=ref('marketo__email_templates'), except=var('consistency_test_exclude_metrics', [])) }}
+    from {{ target.schema }}_marketo_prod.marketo__email_templates
+),
+
+dev as (
+    select {{ dbt_utils.star(from=ref('marketo__email_templates'), except=var('consistency_test_exclude_metrics', [])) }}
+    from {{ target.schema }}_marketo_dev.marketo__email_templates
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_leads.sql

@@ -0,0 +1,45 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select {{ dbt_utils.star(from=ref('marketo__leads'), except=var('consistency_test_exclude_metrics', [])) }}
+    from {{ target.schema }}_marketo_prod.marketo__leads
+),
+
+dev as (
+    select {{ dbt_utils.star(from=ref('marketo__leads'), except=var('consistency_test_exclude_metrics', [])) }}
+    from {{ target.schema }}_marketo_dev.marketo__leads
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

models/intermediate/intermediate.yml

@@ -4,92 +4,27 @@ models:
   - name: int_marketo__lead
 
   - name: marketo__bounces__by_sent_email
-    columns:
-      - name: email_send_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__clicks__by_sent_email
-    columns:
-      - name: email_send_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__deliveries__by_sent_email
-    columns:
-      - name: email_send_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__opens__by_sent_email
-    columns:
-      - name: email_send_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__unsubscribes__by_sent_email
-    columns:
-      - name: email_send_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__change_data_details
-    columns:
-      - name: lead_day_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__change_data_pivot
-    columns:
-      - name: lead_day_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__change_data_scd
-    columns:
-      - name: lead_day_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__email_stats__by_campaign
-    columns:
-      - name: campaign_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__email_stats__by_email_template
-    columns:
-      - name: email_template_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__email_stats__by_lead
-    columns:
-      - name: lead_id
-        tests:
-          - not_null
-          - unique
 
   - name: marketo__email_stats__by_program
-    columns:
-      - name: program_id
-        tests:
-          - not_null
-          - unique
 
-  - name: marketo__email_sends_deduped
-    columns:
-      - name: email_send_id
-        tests:
-          - not_null
-          - unique
+  - name: marketo__email_sends_deduped