Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Enable User models by default #17

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/scripts/run_models.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ dbt deps
dbt seed --target "$db" --full-refresh
dbt run --target "$db" --full-refresh
dbt test --target "$db"
dbt run --vars {'servicenow__using_roles: True'} --target "$db"
dbt run --vars {'servicenow__using_roles: False'} --target "$db"
dbt test --target "$db"

dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
# dbt_servicenow v0.4.0
[PR #17](https://github.com/fivetran/dbt_servicenow/pull/17) includes the following updates:

## Breaking Changes
- Enabled `servicenow__user_aggregated` and `servicenow__user_enhanced` by default by changing the default `servicenow__using_roles` value to true.
- By setting the `servicenow__using_roles` variable to true, we now also enable the upstream `stg_servicenow_*` models that flow into these user tables, which derive from the `sys_user_grmember`, `sys_user_has_role`, and `sys_user_role` source tables.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to clearly callout what new models will be materialized in the warehouse by default. Can you add onto this entry sub-bullets which list out all the new models which will be enabled by default.

Additionally, we should mention that Quickstart will dynamically handle the enable/disabling based on the existence of the source tables or via the Quickstart data model selection tab. However, non Quickstart users will be required to define the variable themselves in their dbt_project.yml if they do not have the required source tables.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated!

- Because this change will introduce new end model tables to users because they were initially disabled by default, we've classified this as a breaking change.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also a breaking change because we are changing the default behavior of a variable. Please also include here that users who do not have these source tables will need to define the variable in their dbt_project.yml

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated!


## Documentation Update
- Updated the variable configuration section of the README since `servicenow__using_roles` is now set to true by default.
- Moved badges at top of the README below the H1 header to be consistent with popular README formats.

## Under the Hood
- Changed Buildkite scripts to run models when `servicenow__using_roles` is set to False (since it's now True by default on all models).

# dbt_servicenow v0.3.0
[PR #10](https://github.com/fivetran/dbt_servicenow/pull/10) includes the following updates:

Expand Down
16 changes: 7 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
<p align="center">
# ServiceNow dbt Package ([Docs](https://fivetran.github.io/dbt_servicenow/))

<p align="left">
<a alt="License"
href="https://github.com/fivetran/dbt_servicenow/blob/main/LICENSE">
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
Expand All @@ -10,9 +12,6 @@
<img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
</p>


# ServiceNow dbt Package ([Docs](https://fivetran.github.io/dbt_servicenow/))

## What does this dbt package do?

This package models ServiceNow data from [Fivetran's connector](https://fivetran.com/docs/applications/servicenow). It uses data in the format described by [this ERD](https://fivetran.com/docs/applications/servicenow#schemainformation).
Expand Down Expand Up @@ -73,7 +72,7 @@ Include the following ServiceNow package version in your `packages.yml` file:
```yml
packages:
- package: fivetran/servicenow
version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically
```

### Step 3: Define database and schema variables
Expand Down Expand Up @@ -104,14 +103,14 @@ To connect your multiple schema/database sources to the package models, follow t

### (Optional) Step 4: Additional configurations

#### Populating the User Models
Our user grain models are disabled by default because not everyone syncs the underlying tables: `sys_user_role`, `sys_user_has_role`, and `sys_user_grmember`. If these tables do exist in your schema, set this following variable `servicenow__using_roles` to True in order to populate the user models `servicenow__user_aggregated` and `servicenow__user_enhanced`.
#### Enable/Disable the User End Models
Our user grain models are enabled by default, but we do understand that not everyone syncs the underlying tables: `sys_user_role`, `sys_user_has_role`, and `sys_user_grmember`. If these tables do not exist in your schema, set this following variable `servicenow__using_roles` to False in order to disable the end models `servicenow__user_aggregated` and `servicenow__user_enhanced`.

```yml
# dbt_project.yml

vars:
servicenow__using_roles: True
servicenow__using_roles: False # Disable if you are not using user roles
```

#### Changing the Build Schema
Expand Down Expand Up @@ -140,7 +139,6 @@ vars:
```
</details>


### (Optional) Step 5: Orchestrate your models with Fivetran Transformations for dbt Core™
<details><summary>Expand for details</summary>
<br>
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
config-version: 2
name: 'servicenow'
version: '0.3.0'
version: '0.4.0'

require-dbt-version: [">=1.3.0", "<2.0.0"]

Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

274 changes: 211 additions & 63 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion docs/run_results.json

This file was deleted.

4 changes: 1 addition & 3 deletions integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'servicenow_integration_tests'
version: '0.3.0'
version: '0.4.0'
config-version: 2

profile: 'integration_tests'
Expand Down Expand Up @@ -27,8 +27,6 @@ vars:
servicenow_sys_user_has_role_identifier: "sys_user_has_role_data"
servicenow_sys_user_grmember_identifier: "sys_user_grmember_data"

# servicenow__using_roles: true # to be turned on when generating docs so that can be generated via the manifest and docs as it is turned off by default.
fivetran-joemarkiewicz marked this conversation as resolved.
Show resolved Hide resolved

models:
+schema: "servicenow_{{ var('directed_schema','dev') }}"

Expand Down
2 changes: 1 addition & 1 deletion models/servicenow__user_aggregated.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}
fivetran-joemarkiewicz marked this conversation as resolved.
Show resolved Hide resolved

with sys_user as (

Expand Down
2 changes: 1 addition & 1 deletion models/servicenow__user_enhanced.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

with user_aggregates as (

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

{{
fivetran_utils.union_data(
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

{{
fivetran_utils.union_data(
Expand Down
2 changes: 1 addition & 1 deletion models/staging/base/stg_servicenow__sys_user_role_base.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

{{
fivetran_utils.union_data(
Expand Down
2 changes: 1 addition & 1 deletion models/staging/stg_servicenow__sys_user_grmember.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

with base as (

Expand Down
2 changes: 1 addition & 1 deletion models/staging/stg_servicenow__sys_user_has_role.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

with base as (

Expand Down
2 changes: 1 addition & 1 deletion models/staging/stg_servicenow__sys_user_role.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('servicenow__using_roles', False)) }}
{{ config(enabled=var('servicenow__using_roles', True)) }}

with base as (

Expand Down