Merge pull request #10 from fivetran/feature/add_display_values

Updates to display names

CHANGELOG.md

@@ -1,3 +1,59 @@
+# dbt_servicenow v0.3.0
+[PR #10](https://github.com/fivetran/dbt_servicenow/pull/10) includes the following updates:
+
+## Breaking Changes
+-  The following fields were removed from the `servicenow__task_enhanced` model as `problem_first_reported_by_task_value` was incorrectly assumed to have corresponding `sys_user` values, but in actuality is related to the task table and therefore have no user fields.
+    - `problem_reporter_email`
+    - `problem_reporter_manager_value`
+    - `problem_reporter_department_value`
+    - `problem_reporter_name`
+    - `problem_reporter_roles`
+
+## Feature Updates
+#### Display Names
+- The following fields were added to the `servicenow__incident_enhanced` model to add relevant display names to the `caller_id_value`.
+  - `caller_email` 
+  - `caller_manager_value`
+  - `caller_department_value`
+  - `caller_name`
+  - `caller_roles`
+- The following fields were added to the `servicenow__user_enhanced` model to add relevant display names to the `company_value` and `manager_value`. 
+  - `company_name`
+  - `manager_name`
+- The following label fields were added to respective choice fields in order to bring the display value (`dv`), or human-readable text available in the ServiceNow UI, displayed for each choice option:
+  - `dv_priority_label`
+  - `dv_impact_label`
+  - `dv_urgency_label`
+  - `dv_task_state_label`
+  - `dv_problem_state_label`
+  - `dv_change_request_type_label`
+  - `dv_change_request_phase_state_label`
+  - `dv_change_request_risk_label`
+  - `dv_incident_category_label`
+  - `dv_incident_subcategory_label`
+  - `dv_incident_severity_label`
+  - `dv_incident_state_label`
+  - `dv_business_impact_label`
+
+- Certain choice fields have labels that can depend on the `dependent_value` field in the `sys_choice` table. This dependency typically arises when the same choice field can have different labels based on the context in which it is used. Therefore some elements from `sys_choice` are also joined in on the additional `dependent_value` field. For example, here is a common example where `dependent_value` might influence the label:
+    - `incident_category` and `incident_subcategory`: These fields often depend on each other. For instance, if the `incident_category` is "Software," the `incident_subcategory` might have different options compared to when the category is "Hardware." The `dependent_value` in this case could represent the incident_category and influence the incident_subcategory labels.
+    - For more information about the logic used to attach labels to choice fields, refer to the [DECISIONLOG](https://github.com/fivetran/dbt_servicenow/blob/main/DECISIONLOG.md#methodology-for-adding-label-for-choice-fields) where a section has been added (titled *Methodology for Adding Labels for Choice Fields*).
+
+#### New Staging Model
+- To include labels, we used a new source table `sys_choice`. This is reflected upstream, in the newly added `stg_servicenow__sys_choice` and `stg_servicenow__sys_choice_base`. 
+
+#### Aggregations in User Models
+- The following fields, which previously only existed in intermediate CTEs, have been persisted through to the `servicenow__user_aggregated` and `servicenow__user_enhanced` models to add more user metadata:
+  - `count_distinct_sys_user_group_ids`
+  - `count_distinct_sys_user_role_ids`
+  - `count_distinct_sys_user_role_names`
+  - `count_distinct_included_roles`
+  - `count_distinct_sys_user_group_roles`
+- Additionally, we've realized the way these fields were previously aggregated could unintentionally also include blank values that were not-null, so we have updated the logic accordingly.
+
+## Under The Hood
+- Added explicit casts to timestamp fields, as well as string casts to choice fields in order to join them later downstream on `sys_choice.element`.
+- Added consistency and integrity validation tests within integration_tests folder for all current end models.
 
 # dbt_servicenow v0.2.0
 [PR #8](https://github.com/fivetran/dbt_servicenow/pull/8) includes the following updates:

DECISIONLOG.md

@@ -1,3 +1,11 @@
+## Methodology for Adding Labels for Choice Fields 
+
+A choice field is a type of field that allows users to select from a predefined list of options (for example, `incident_subcategory` and `change_request_risk`). These options are stored in the `sys_choice` table, which contains the possible values, labels, and other metadata for the choice field. Each choice field is typically associated with a specific table and column in the database and is used to standardize the input for commonly selected values, such as incident states, categories, or priorities.
+
+Certain choice fields have labels that can depend on the `dependent_value` field in the `sys_choice` table. This dependency typically arises when the same choice field can have different labels based on the context in which it is used. Therefore some elements from `sys_choice` are also joined in on additional fields. Here are some common examples where `dependent_value` might influence the label:
+
+- `incident_category` and `incident_subcategory`: These fields often depend on each other. For instance, if the `incident_category` is "Software," the `incident_subcategory` might have different options compared to when the category is "Hardware." The `dependent_value` in this case could represent the `incident_category` and influence the `incident_subcategory` labels.
+
 ## Relationships between User, Group, and Role related tables in ServiceNow
 
 The Users table (`sys_user`) is linked to Roles (`sys_user_role`) through a many-to-many relationship via the `sys_user_has_role` table. This intermediary table stores records that connect each User with one or more Roles. When a Role is assigned directly to a User, a corresponding entry is created in the `sys_user_has_role` table. Conversely, when a Role is removed from a User, the corresponding entry is deleted from the table.

README.md

@@ -73,7 +73,7 @@ Include the following ServiceNow package version in your `packages.yml` file:
 ```yml
 packages:
   - package: fivetran/servicenow
-    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ## Step 3: Define database and schema variables

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'servicenow'
-version: '0.2.0'
+version: '0.3.0'
 
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
@@ -26,6 +26,7 @@ vars:
     core_company: "{{ source('servicenow','core_company') }}"
     cmdb_ci: "{{ source('servicenow','cmdb_ci') }}"
     cmdb_ci_service: "{{ source('servicenow','cmdb_ci_service') }}"
+    sys_choice: "{{ source('servicenow','sys_choice') }}"
     sys_user_group: "{{ source('servicenow','sys_user_group') }}"
     sys_user_role: "{{ source('servicenow','sys_user_role') }}"
     sys_user_has_role: "{{ source('servicenow','sys_user_has_role') }}"

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_13
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_13
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_13
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_13
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_13
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'servicenow_integration_tests'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 
 profile: 'integration_tests'
@@ -10,7 +10,7 @@ dispatch:
 
 vars:
   servicenow:
-    servicenow_schema: servicenow_integration_tests_10
+    servicenow_schema: servicenow_integration_tests_13
     servicenow_task_identifier: "task_data"
     servicenow_task_sla_identifier: "task_sla_data"
     servicenow_sys_user_identifier: "sys_user_data"
@@ -27,6 +27,10 @@ vars:
     servicenow_sys_user_has_role_identifier: "sys_user_has_role_data"
     servicenow_sys_user_grmember_identifier: "sys_user_grmember_data"
 
+    # servicenow__using_roles: true # to be turned on when generating docs so that can be generated via the manifest and docs as it is turned off by default.
+
+models:
+  +schema: "servicenow_{{ var('directed_schema','dev') }}"
 
 seeds:
   +quote_columns: "{{ true if target.type in ('redshift','postgres') else false }}"
@@ -72,4 +76,4 @@ seeds:
       +enabled: "{{ true if target.type != 'snowflake' else false }}"
     task_data_snowflake:
       +alias: task_data
-      +enabled: "{{ true if target.type == 'snowflake' else false }}"
+      +enabled: "{{ true if target.type == 'snowflake' else false }}"

integration_tests/tests/consistency/activity_enhanced_aggs.sql

@@ -0,0 +1,84 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        task_updated_date,
+        source_relation,
+        sum(total_active_tasks) as total_active_tasks,
+        sum(total_active_problem_tasks) as total_active_problem_tasks,
+        sum(total_active_change_tasks) as total_active_change_tasks,
+        sum(total_associated_problems) as total_associated_problems, 
+        sum(total_incidents_caused_by_problems) as total_incidents_caused_by_problems,
+        sum(total_change_requests) as total_change_requests,
+        sum(total_closed_tasks) as total_closed_tasks,
+        sum(average_minutes_open_to_close) as average_minutes_open_to_close,
+        sum(total_tasks_made_slas) as total_tasks_made_slas
+
+    from {{ target.schema }}_servicenow_prod.servicenow__activity_summary
+    group by 1,2
+),
+
+dev as (
+    select
+        task_updated_date,
+        source_relation,
+        sum(total_active_tasks) as total_active_tasks,
+        sum(total_active_problem_tasks) as total_active_problem_tasks,
+        sum(total_active_change_tasks) as total_active_change_tasks,
+        sum(total_associated_problems) as total_associated_problems, 
+        sum(total_incidents_caused_by_problems) as total_incidents_caused_by_problems,
+        sum(total_change_requests) as total_change_requests,
+        sum(total_closed_tasks) as total_closed_tasks,
+        sum(average_minutes_open_to_close) as average_minutes_open_to_close,
+        sum(total_tasks_made_slas) as total_tasks_made_slas
+
+    from {{ target.schema }}_servicenow_dev.servicenow__activity_summary
+    group by 1,2
+),
+
+final as (
+    select
+        prod.task_updated_date as prod_task_updated_date,
+        dev.task_updated_date as dev_task_updated_date,
+        prod.source_relation as prod_source_relation,
+        dev.source_relation as dev_source_relation,
+        prod.total_active_tasks as prod_total_active_tasks,
+        dev.total_active_tasks as dev_total_active_tasks,
+        prod.total_active_problem_tasks as prod_total_active_problem_tasks,
+        dev.total_active_problem_tasks as dev_total_active_problem_tasks,
+        prod.total_active_change_tasks as prod_total_active_change_tasks,
+        dev.total_active_change_tasks as dev_total_active_change_tasks,
+        prod.total_associated_problems as prod_total_associated_problems, 
+        dev.total_associated_problems as dev_total_associated_problems, 
+        prod.total_incidents_caused_by_problems as prod_total_incidents_caused_by_problems,
+        dev.total_incidents_caused_by_problems as dev_total_incidents_caused_by_problems,
+        prod.total_change_requests as prod_total_change_requests,
+        dev.total_change_requests as dev_total_change_requests,
+        prod.total_closed_tasks as prod_total_closed_tasks,
+        dev.total_closed_tasks as dev_total_closed_tasks,
+        prod.average_minutes_open_to_close as prod_average_minutes_open_to_close,
+        dev.average_minutes_open_to_close as dev_average_minutes_open_to_close,
+        prod.total_tasks_made_slas as prod_total_tasks_made_slas,
+        dev.total_tasks_made_slas as dev_total_tasks_made_slas
+
+    from prod
+    full outer join dev
+        on prod.task_updated_date = dev.task_updated_date
+        and prod.source_relation = dev.source_relation
+)
+
+select * 
+from final
+where
+    abs(prod_total_active_tasks - dev_total_active_tasks) > .1 or
+    abs(prod_total_active_problem_tasks - dev_total_active_problem_tasks) > .1 or
+    abs(prod_total_active_change_tasks - dev_total_active_change_tasks) > .1 or
+    abs(prod_total_associated_problems - dev_total_associated_problems) > .1 or
+    abs(prod_total_incidents_caused_by_problems - dev_total_incidents_caused_by_problems) > .1 or
+    abs(prod_total_change_requests - dev_total_change_requests) > .1 or
+    abs(prod_total_closed_tasks - dev_total_closed_tasks) > .1 or
+    abs(prod_average_minutes_open_to_close - dev_average_minutes_open_to_close) > .1 or
+    abs(prod_total_tasks_made_slas - dev_total_tasks_made_slas) > .1

integration_tests/tests/consistency/change_request_enhanced_row_counts.sql

@@ -0,0 +1,38 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        1 as join_key,
+        source_relation,
+        count(*) as row_count
+    from {{ target.schema }}_servicenow_prod.servicenow__change_request_enhanced
+    group by 1,2
+),
+
+dev as (
+    select
+        1 as join_key,
+        source_relation,
+        count(*) as row_count
+
+    from {{ target.schema }}_servicenow_dev.servicenow__change_request_enhanced
+    group by 1,2
+),
+
+final as (
+    select 
+        prod.row_count as prod_row_count,
+        dev.row_count as dev_row_count
+
+    from prod
+    full outer join dev
+        on prod.join_key = dev.join_key
+        and prod.source_relation = dev.source_relation
+)
+
+select * 
+from final
+where prod_row_count != dev_row_count

integration_tests/tests/consistency/incident_enhanced_row_counts.sql

@@ -0,0 +1,38 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        1 as join_key,
+        source_relation,
+        count(*) as row_count
+    from {{ target.schema }}_servicenow_prod.servicenow__incident_enhanced
+    group by 1,2
+),
+
+dev as (
+    select
+        1 as join_key,
+        source_relation,
+        count(*) as row_count
+
+    from {{ target.schema }}_servicenow_dev.servicenow__incident_enhanced
+    group by 1,2
+),
+
+final as (
+    select 
+        prod.row_count as prod_row_count,
+        dev.row_count as dev_row_count
+
+    from prod
+    full outer join dev
+        on prod.join_key = dev.join_key
+        and prod.source_relation = dev.source_relation
+)
+
+select * 
+from final
+where prod_row_count != dev_row_count

integration_tests/tests/consistency/problem_enhanced_row_counts.sql

@@ -0,0 +1,38 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        1 as join_key,
+        source_relation,
+        count(*) as row_count
+    from {{ target.schema }}_servicenow_prod.servicenow__problem_enhanced
+    group by 1,2
+),
+
+dev as (
+    select
+        1 as join_key,
+        source_relation,
+        count(*) as row_count
+
+    from {{ target.schema }}_servicenow_dev.servicenow__problem_enhanced
+    group by 1,2
+),
+
+final as (
+    select 
+        prod.row_count as prod_row_count,
+        dev.row_count as dev_row_count
+
+    from prod
+    full outer join dev
+        on prod.join_key = dev.join_key
+        and prod.source_relation = dev.source_relation
+)
+
+select * 
+from final
+where prod_row_count != dev_row_count