Report enhancements

CHANGELOG.md

@@ -1,3 +1,20 @@
+# dbt_servicenow v0.3.0
+[PR #10](https://github.com/fivetran/dbt_servicenow/pull/10) includes the following updates:
+
+## 🚨 Breaking Changes
+-  The following fields were removed from the `servicenow__task_enhanced` model as `problem_first_reported_by_task_value` was incorrectly assumed to have corresponding `sys_user` values, but in actuality is related to the task table and therefore have no user fields.
+  - `problem_reporter_email, problem_reporter_manager_value, problem_reporter_department_value, problem_reporter_name, problem_reporter_roles`
+
+## Feature Updates
+- The following fields were added to the `servicenow__incident_enhanced` model to add relevant display names to the caller_id_value.
+  - `caller_email, caller_manager_value, caller_department_value, caller_name, caller_roles`
+- The following fields were added to the `servicenow__user_enhanced` model to add relevant display names to the company_value and manager_value. 
+  - `company_name, manager_name`
+- The following label fields were added to respective choice fields in order to bring the human-readable text displayed for each choice option:
+  - `dv_priority_label, dv_impact_label, dv_urgency_label, dv_task_state_label, dv_problem_state_label, dv_change_request_type_label, dv_change_request_phase_state_label, dv_change_request_risk_label, dv_risk_impact_analysis_label, dv_incident_category_label, dv_incident_subcategory_label, dv_incident_severity_label, dv_incident_state_label, dv_business_impact_label`
+    - Certain choice fields have labels that can depend on the `dependent_value` field in the `sys_choice` table. This dependency typically arises when the same choice field can have different labels based on the context in which it is used. Therefore some elements from `sys_choice` are also joined in on the additional `dependent_value` field. For example, here is a common example where `dependent_value` might influence the label:
+
+    - `incident_category` and `incident_subcategory`: These fields often depend on each other. For instance, if the `incident_category` is "Software," the `incident_subcategory` might have different options compared to when the category is "Hardware." The `dependent_value` in this case could represent the incident_category and influence the incident_subcategory labels.
 
 # dbt_servicenow v0.2.0
 [PR #8](https://github.com/fivetran/dbt_servicenow/pull/8) includes the following updates:

DECISIONLOG.md

@@ -1,3 +1,11 @@
+## Methodology for Adding Labels for Choice Fields 
+
+A choice field is a type of field that allows users to select from a predefined list of options (for example, `incident_subcategory` and `change_request_risk`). These options are stored in the `sys_choice` table, which contains the possible values, labels, and other metadata for the choice field. Each choice field is typically associated with a specific table and column in the database and is used to standardize the input for commonly selected values, such as incident states, categories, or priorities.
+
+Certain choice fields have labels that can depend on the `dependent_value` field in the `sys_choice` table. This dependency typically arises when the same choice field can have different labels based on the context in which it is used. Therefore some elements from `sys_choice` are also joined in on additional fields. Here are some common examples where `dependent_value` might influence the label:
+
+- `incident_category` and `incident_subcategory`: These fields often depend on each other. For instance, if the `incident_category` is "Software," the `incident_subcategory` might have different options compared to when the category is "Hardware." The `dependent_value` in this case could represent the `incident_category` and influence the `incident_subcategory` labels.
+
 ## Relationships between User, Group, and Role related tables in ServiceNow
 
 The Users table (`sys_user`) is linked to Roles (`sys_user_role`) through a many-to-many relationship via the `sys_user_has_role` table. This intermediary table stores records that connect each User with one or more Roles. When a Role is assigned directly to a User, a corresponding entry is created in the `sys_user_has_role` table. Conversely, when a Role is removed from a User, the corresponding entry is deleted from the table.

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'servicenow'
-version: '0.2.0'
+version: '0.3.0'
 
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
@@ -26,6 +26,7 @@ vars:
     core_company: "{{ source('servicenow','core_company') }}"
     cmdb_ci: "{{ source('servicenow','cmdb_ci') }}"
     cmdb_ci_service: "{{ source('servicenow','cmdb_ci_service') }}"
+    sys_choice: "{{ source('servicenow','sys_choice') }}"
     sys_user_group: "{{ source('servicenow','sys_user_group') }}"
     sys_user_role: "{{ source('servicenow','sys_user_role') }}"
     sys_user_has_role: "{{ source('servicenow','sys_user_has_role') }}"

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_11
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_11
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_11
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_11
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: servicenow_integration_tests_10
+      schema: servicenow_integration_tests_11
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'servicenow_integration_tests'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 
 profile: 'integration_tests'
@@ -10,7 +10,7 @@ dispatch:
 
 vars:
   servicenow:
-    servicenow_schema: servicenow_integration_tests_10
+    servicenow_schema: servicenow_integration_tests_11
     servicenow_task_identifier: "task_data"
     servicenow_task_sla_identifier: "task_sla_data"
     servicenow_sys_user_identifier: "sys_user_data"

macros/get_sys_choice_columns.sql

@@ -0,0 +1,29 @@
+{% macro get_sys_choice_columns() %}
+
+{% set columns = [
+    {"name": "sys_id", "datatype": dbt.type_string()},
+    {"name": "_fivetran_deleted", "datatype": dbt.type_boolean()},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "dependent_value", "datatype": dbt.type_string()},
+    {"name": "element", "datatype": dbt.type_string()},
+    {"name": "hint", "datatype": dbt.type_string()},
+    {"name": "inactive", "datatype": dbt.type_boolean()},
+    {"name": "label", "datatype": dbt.type_string()},
+    {"name": "language", "datatype": dbt.type_string()},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "sequence", "datatype": dbt.type_int()},
+    {"name": "synonyms", "datatype": dbt.type_string()},
+    {"name": "sys_created_by", "datatype": dbt.type_string()},
+    {"name": "sys_created_on", "datatype": dbt.type_timestamp()},
+    {"name": "sys_domain_link", "datatype": dbt.type_string()},
+    {"name": "sys_domain_path", "datatype": dbt.type_string()},
+    {"name": "sys_domain_value", "datatype": dbt.type_string()},
+    {"name": "sys_mod_count", "datatype": dbt.type_int()},
+    {"name": "sys_updated_by", "datatype": dbt.type_string()},
+    {"name": "sys_updated_on", "datatype": dbt.type_timestamp()},
+    {"name": "value", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

models/docs.md

@@ -1931,3 +1931,51 @@ List of included roles.
 {% docs sys_user_group_roles %}
 List of associated user group roles.
 {% enddocs %}
+
+{% docs sys_choice %}
+Staging table containing all records relating to system choices. System choices represent predefined options for fields in various ServiceNow tables.
+{% enddocs %}
+
+{% docs stg_servicenow__sys_choice %}
+Staging table containing all records relating to system choices. System choices represent predefined options for fields in various ServiceNow tables.
+{% enddocs %}
+
+{% docs dependent_value %}
+The value of another field that this choice depends on. Used for creating dependent choice lists.
+{% enddocs %}
+
+{% docs element %}
+The name of the field that this choice is associated with, for example Urgency.
+{% enddocs %}
+
+{% docs hint %}
+Additional information or guidance about the choice, often displayed as a tooltip or helper text.
+{% enddocs %}
+
+{% docs inactive %}
+Boolean indicating whether this choice is inactive (not available for selection).
+{% enddocs %}
+
+{% docs label %}
+The human-readable text displayed for this choice option.
+{% enddocs %}
+
+{% docs language %}
+The language code for this choice, used for multilingual implementations.
+{% enddocs %}
+
+{% docs sequence %}
+A number indicating the order in which this choice should appear in a list.
+{% enddocs %}
+
+{% docs synonyms %}
+Alternative terms or phrases that are equivalent to this choice, used for searching or matching.
+{% enddocs %}
+
+{% docs sys_choice_value %}
+The stored value for this choice option, often used in database operations or scripts. The actual value stored in the element. (e.g., for element = Urgency: 1, 2, or 3).
+{% enddocs %}
+
+{% docs sys_choice_name %}
+Indicates the table to which the field belongs, such as task.
+{% enddocs %}

models/servicenow.yml

@@ -232,6 +232,9 @@ models:
       - name: problem_confirmed_by_value
         description: '{{ doc("problem_confirmed_by_value") }}'
 
+      - name: problem_first_reported_by_task_link
+        description: '{{ doc("problem_first_reported_by_task_link") }}'
+
       - name: problem_first_reported_by_task_value
         description: '{{ doc("task_value") }}'
 
@@ -313,7 +316,7 @@ models:
       - name: change_request_category
         description: '{{ doc("change_category") }}'
 
-      - name: change_plan
+      - name: change_request_change_plan
         description: '{{ doc("change_plan") }}'
 
       - name: change_request_close_code
@@ -379,6 +382,18 @@ models:
       - name: is_change_request_unauthorized
         description: '{{ doc("is_change_request_unauthorized") }}'
 
+      - name: dv_priority_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_impact_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_urgency_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_task_state_label
+        description: '{{ doc("label") }}'
+
       - name: is_problem_task
         description: '{{ doc("is_problem_task") }}'
       - name: is_change_task
@@ -455,16 +470,6 @@ models:
         description: '{{ doc("sys_user_name") }}'
       - name: problem_confirmer_roles
         description: '{{ doc("roles") }}'
-      - name: problem_reporter_email
-        description: '{{ doc("email") }}'
-      - name: problem_reporter_manager_value
-        description: '{{ doc("manager_value") }}'
-      - name: problem_reporter_department_value
-        description: '{{ doc("department_value") }}'
-      - name: problem_reporter_name
-        description: '{{ doc("sys_user_name") }}'
-      - name: problem_reporter_roles
-        description: '{{ doc("roles") }}'
       - name: problem_fixer_email
         description: '{{ doc("email") }}'
       - name: problem_fixer_manager_value
@@ -770,6 +775,9 @@ models:
         description: '{{ doc("sys_user_name") }}'
       - name: problem_workaround_communicator_roles
         description: '{{ doc("roles") }}'
+      - name: dv_problem_state_label
+        description: '{{ doc("label") }}'
+
 
 
   - name: servicenow__change_request_enhanced
@@ -826,7 +834,7 @@ models:
       - name: change_plan
         description: '{{ doc("change_plan") }}'
 
-      - name: backout_plan
+      - name: change_request_backout_plan
         description: '{{ doc("backout_plan") }}'
 
       - name: chg_model_link
@@ -940,6 +948,16 @@ models:
       - name: source_relation
         description: '{{ doc("source_relation") }}'
 
+      - name: dv_change_request_type_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_change_request_phase_state_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_change_request_risk_label
+        description: '{{ doc("label") }}'
+
+
   - name: servicenow__incident_enhanced
     description: '{{ doc("servicenow__incident_enhanced") }}'
     tests:
@@ -1031,6 +1049,21 @@ models:
 
       - name: caller_id_value
         description: '{{ doc("caller_id_value")}}'
+
+      - name: caller_email
+        description: '{{ doc("email")}}'
+
+      - name: caller_manager_value
+        description: '{{ doc("manager_value")}}'
+
+      - name: caller_department_value
+        description: '{{ doc("department_value")}}'
+
+      - name: caller_name
+        description: '{{ doc("sys_user_name")}}'
+
+      - name: caller_roles
+        description: '{{ doc("roles")}}'
 
       - name: incident_cause
         description: '{{ doc("cause")}}'
@@ -1089,6 +1122,22 @@ models:
       - name: source_relation
         description: '{{ doc("source_relation")}}'
 
+      - name: dv_incident_category_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_incident_subcategory_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_incident_severity_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_incident_state_label
+        description: '{{ doc("label") }}'
+
+      - name: dv_business_impact_label
+        description: '{{ doc("label") }}'
+
+
 
   - name: servicenow__user_aggregated
     description: '{{ doc("servicenow__user_aggregated") }}'
@@ -1153,6 +1202,8 @@ models:
         description: '{{ doc("company_link") }}'
       - name: company_value
         description: '{{ doc("company_value") }}'
+      - name: company_name
+        description: '{{ doc("name") }}'
       - name: cost_center_link
         description: '{{ doc("cost_center_link") }}'
       - name: cost_center_value
@@ -1179,6 +1230,8 @@ models:
         description: '{{ doc("manager_link") }}'
       - name: manager_value
         description: '{{ doc("manager_value") }}'
+      - name: manager_name
+        description: '{{ doc("sys_user_name") }}'
       - name: mobile_phone
         description: '{{ doc("mobile_phone") }}'
       - name: phone