Merge pull request #29 from fivetran/feature/databricks-compat

Feature/databricks compat
fivetran · Nov 14, 2022 · 056c4e5 · 056c4e5
2 parents 5f23ac5 + d05653d
commit 056c4e5
Show file tree

Hide file tree

Showing 12 changed files with 45 additions and 11 deletions.
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
@@ -42,3 +42,17 @@ steps:
             - "CI_REDSHIFT_DBT_USER"
     commands: |
       bash .buildkite/scripts/run_models.sh redshift
+
+  - label: ":bricks: Run Tests - Databricks"
+    key: "run_dbt_databricks"
+    plugins:
+      - docker#v3.13.0:
+          image: "python:3.8"
+          shell: [ "/bin/bash", "-e", "-c" ]
+          environment:
+            - "BASH_ENV=/tmp/.bashrc"
+            - "CI_DATABRICKS_DBT_HOST"
+            - "CI_DATABRICKS_DBT_HTTP_PATH"
+            - "CI_DATABRICKS_DBT_TOKEN"
+    commands: |
+      bash .buildkite/scripts/run_models.sh databricks
diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+set -euo pipefail
+
 apt-get update
 apt-get install libsasl2-dev
 

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,4 +1,3 @@
-Pull Request
 **Are you a current Fivetran customer?** 
 <!--- Please tell us your name, title and company -->
 
@@ -29,9 +28,9 @@ Pull Request
 
 **How did you test the PR changes?** 
 <!--- Proof of testing is required in order for the PR to be approved. -->
-<!--- To check a box, remove the space and insert an x in the box (eg. [x] CircleCi). --> 
+<!--- To check a box, remove the space and insert an x in the box (eg. [x] Buildkite). --> 
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] CircleCi <!--- CircleCi testing is only applicable to Fivetran employees. --> 
+- [ ] Buildkite <!--- Buildkite testing is only applicable to Fivetran employees. --> 
 - [ ] Local (please provide additional testing details below)
 
 **Select which warehouse(s) were used to test the PR**

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -37,8 +37,9 @@
 - Dependencies on `fivetran/fivetran_utils` have been upgraded, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
 
 ## 🎉 Documentation and Feature Updates 🎉:
-- Updated README documentation for easier navigation and dbt package setup.
-- Included the `github_[source_table_name]_identifier` variables for easier flexibility of the package models to refer to differently named sources tables.
+- Updated README documentation for easier navigation and dbt package setup [#26](https://github.com/fivetran/dbt_github_source/pull/26).
+- Included the `github_[source_table_name]_identifier` variables for easier flexibility of the package models to refer to differently named sources tables [#26](https://github.com/fivetran/dbt_github_source/pull/26).
+- Added Databricks compatibility [#29](https://github.com/fivetran/dbt_github_source/pull/29).
 
 # dbt_github_source v0.5.0
 

diff --git a/README.md b/README.md
@@ -24,7 +24,15 @@
 ## Step 1: Prerequisites
 To use this dbt package, you must have the following:
 - At least one Fivetran Github connector syncing data into your destination. 
-- A **BigQuery**, **Snowflake**, or **Redshift** destination.
+- A **BigQuery**, **Snowflake**, **Redshift** or **Databricks**  destination.
+
+### Databricks Dispatch Configuration
+If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively.
+```yml
+dispatch:
+  - macro_namespace: dbt_utils
+    search_order: ['spark_utils', 'dbt_utils']
+```
 
 ## Step 2: Install the package
 Include the following github_source package version in your `packages.yml` file.
@@ -95,6 +103,10 @@ packages:
 
     - package: dbt-labs/dbt_utils
       version: [">=1.0.0", "<2.0.0"]
+
+    - package: dbt-labs/spark_utils
+      version: [">=0.3.0", "<0.4.0"]
+
 ```
 # 🙌 How is this package maintained and can I contribute?
 ## Package Maintenance

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/ci/sample.profiles.yml b/integration_tests/ci/sample.profiles.yml
@@ -1,5 +1,5 @@
 
-# HEY! This file is used in the dbt package integrations tests with CircleCI.
+# HEY! This file is used in the dbt package integrations tests with Buildkite.
 # You should __NEVER__ check credentials into version control. Thanks for reading :)
 
 config:

diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -55,3 +55,7 @@ seeds:
     github_requested_reviewer_history_data:
       +column_types:
         created_at: timestamp
+
+dispatch:
+  - macro_namespace: dbt_utils
+    search_order: ['spark_utils', 'dbt_utils']
diff --git a/models/src_github.yml b/models/src_github.yml
@@ -2,7 +2,7 @@ version: 2
 
 sources:
   - name: github
-    database: "{{var ('github_database', target.database)}}"
+    database: "{% if target.type != 'spark'%}{{ var('github_database', target.database) }}{% endif %}"
     schema: "{{var ('github_schema', 'github')}}"
     loader: fivetran
     loaded_at_field: _fivetran_synced

diff --git a/packages.yml b/packages.yml
@@ -1,4 +1,6 @@
 packages:
 - package: fivetran/fivetran_utils
   version: [">=0.4.0", "<0.5.0"]
+- package: dbt-labs/spark_utils
+  version: [">=0.3.0", "<0.4.0"]