Grai Actions

Shared Fields

All actions share some common fields listed below.

Authentication

Field	Required	Description
api-key	no	Your Grai API key.
grai-user	no	Your Grai username.
grai-password	no	Your Grai password.
workspace	no	Your Grai workspace name

You must provider either api-key or grai-user and grai-password. If you're account is associated with multiple workspaces and you're using username/password authentication you must also provide your desired workspace.

Other Parameters

Field	Required	Default	Description
namespace	yes		The Grai namespace for the connection
grai-api-url	no	`https://api.grai.io`	"The url of your grai instance. This is constructed as {scheme}://{host}:{port} where the port is optional"
grai-app-url	no	`https://app.grai.io`	The URL for your frontend instance of Grai. This might include a port depending on your configuration
action	no	`tests`	Which action to perform. Can be `tests` or `update`
github-token	no	`${{github.token}}`	The GITHUB_TOKEN secret for your repository

Notes and Caveats

Github Authentication

By default we use a github-token provided by your repository to write comments back to your PR with test results. In some cases, such as when the pull request is coming from a forked repository, the default token will not have write permissions. If this is the case, you'll receive an error message in the workflow indicating such. There are a few ways you can resolve the issue but you should first check your repository action settings under Settings -> Actions -> General aren't blocking workflows from running.

Some alternatives include.

Explicit Workflow Permissions

GitHub has provided helpful documentation to provide explicit permissions for your workflows. Make sure the Grai Action has, at minimum, write permissions for pull-request and issues. You can set this at the job level by adding a permission key in your workflow. e.g.

jobs:
  my-grai-action:
    runs-on: ubuntu-latest

    permissions:
      issues: write
      pull-requests: write

Personal Access Tokens

You can also use personal access tokens or PAT's in place of the default github-token. You'll need to create a token following the linked instructions but make sure to store it in your repository secrets Settings -> Secrets and variables -> Actions -> New Repository Secret. If you were to create a secret called MY_PAT you would pass it into your grai action job as

jobs:
  my-grai-action:
    runs-on: ubuntu-latest

  steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Run Grai Action
      uses: grai-io/grai-actions/redshift@master
      with:
        github-token: ${{ secrets.MY_PAT }}

Snowflake

The Snowflake action depends on Snowflake's python connector library. You can find complete documentation about the library in the Snowflake docs here with more detail about the connector here.

Fields

Field	Required	Description
db-user	yes	The database user
db-password	yes	The database password
account	yes	Associated Snowflake account
warehouse	yes	Associated Snowflake warehouse
role	no	Optional Snowflake role
database	no	Optional Snowflake database
schema	no	Optional snowflake schema

Example

on:
  - pull_request
name: Snowflake
jobs:
  test_snowflake:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/snowflake@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          db-user: my-user
          db-password: my-password
          account: my-account
          warehouse: my-warehouse

Redshift

The Redshift action depends on Amazon's python connector library. You can find complete documentation about the library in the AWS docs here.

Fields

Field	Required	Default	Description
db-host	yes		The database host
db-port	no	5439	The database port
db-database-name	yes		The database name
db-user	yes		The database user
db-password	yes		The database password

Example

on:
  - pull_request
name: Redshift
jobs:
  test_redshift:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/redshift@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          db-host: redshift-cluster-1.abc123xyz789.us-east-1.redshift.amazonaws.com
          db-port: '5439'
          db-database-name: dev
          db-user: admin
          db-password: password

MySQL

The MySQL action depends on the python mysql library. You can find complete documentation about the library here.

Fields

Field	Required	Default	Description
db-host	yes		The database host
db-port	no	3306	The database port
db-database-name	yes		The database name
db-user	yes		The database user
db-password	yes		The database password

Example

on:
  - pull_request
name: MySQL
jobs:
  test_mysql:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/mysql@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          db-host: dev.mysql.com
          db-port: '3306'
          db-database-name: my_db
          db-user: my_user
          db-password: my_password

Flat File

The flat file action reads a flat file like csv, parquet, or feather inside of your github project to perform tests and update your grai instance. Because of this, it's critical your file is up to date on each pull request.

Make sure to include an - uses: actions/checkout@v3 step in your workflow so that your repo code is available.

Fields

Field	Required	Default	Description
file	yes		Local file to track with Grai

Example

on:
  - pull_request
name: Flat File
jobs:
  test_flat-file:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/flat-file@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          file: ./tests/flat-file/low-numbers.csv

BigQuery

The BigQuery action depends on Google's python BigQuery library. More information can be found about specific connection credentials in Google's documentation here.

Fields

Field	Required	Description
project	yes	The BigQuery project string
dataset	yes	The BigQuery dataset string
credentials	yes	A JSON credential string for use with google oauth service account connections

Example

on:
  - pull_request
name: BigQuery
jobs:
  test_bigquery:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/bigquery@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          project: my-bigquery-project
          dataset: my-bigquery-dataset
          credentials: '{ "type": "service_account", "project_id": "demo", "private_key_id":
            "your_private_key_id", "private_key": "your_private_key", "client_email":
            "[email protected]", "client_id": "your_client_id", "auth_uri":
            "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token",
            "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
            "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/you%40email.iam.gserviceaccount.com"
            }'

Postgres

The Postgres action depends on the python psycopg2 library. You can find complete documentation about the library here.

Fields

Field	Required	Default	Description
db-host	yes		The database host
db-port	no	5432	The database port
db-database-name	yes		The database name
db-user	yes		The database user
db-password	yes		The database password

Example

on:
  - pull_request
name: PostgreSQL
jobs:
  test_postgres:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/postgres@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          db-host: prod.db.com
          db-port: '5432'
          db-database-name: my_database
          db-user: my_user
          db-password: my_password

dbt

The dbt action reads a manifest.json file inside of your github project to perform tests and update your grai instance. Because of this, it's critical your manifest.json file is up to date on each pull request. One way to do this is to perform dbt build as part of your CI action but there are multiple ways to generate an up-to-date manifest file.

More information about manifest.json files can be found here.
Make sure to include an - uses: actions/checkout@v3 step in your workflow so that your repo code is available.

Fields

Field	Required	Default	Description
manifest-file	yes		The file location in your repository of the updated manifest.json file

Example

on:
  - pull_request
name: dbt
jobs:
  test_dbt:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/dbt@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          manifest-file: ./tests/dbt/manifest.json

Fivetran

The Fivetran Action relies upon access to Fivetran's API endpoint. This endpoint is configurable if you have a non-standard implementation but should generally be left alone.

Authentication with their services will require an API key and secret but you can find more documentation about generating these values here.

Fields

Field	Required	Default	Description
fivetran-endpoint	no	https://api.fivetran.com/v1	Fivetran API endpoint
fivetran-api-key	yes		Your Fivetran user api key
fivetran-api-secret	yes		Your Fivetran user api secret
namespace-map	no		A JSON string containing a mapping between Fivetran connections and Grai namespaces

The namespace field in the Fivetran Action works slightly differently than other action. It is used as a default namespace for all connections not specified in the namespace_map. You can find more information about that below.

Namespace Map

Each Fivetran connection has a connector id and synchronizes from a source to a sync. For example, a sync from your production database to data warehouse would have an associated connector id.

Because the Fivetran Action synchronizes from all of your Fivetran connections it uses the namespace_map value to know which connectors belong to which Grai namespaces. The namespace map should be a JSON string with the Grai namespace for each source and destination of each connector id e.g.

{
    "<connector_id>": {
        "source": "<source_namespace>",
        "destination", "<destination_namespace>"
    }
}

You can find connector id's for all of your Fivetran connections in the API

Example

on:
  - pull_request
name: Fivetran
jobs:
  test_fivetran:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/fivetran@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          fivetran-api-key: hHqP5c2nIY0B6fpa
          fivetran-api-secret: 1234567890abcdef1234567890abcdef
          namespace-map: '{"operative_combination": {"source": "source_namespace",
            "destination": "destination_namespace"}}'

Microsoft SQL Server

The SQL Server action depends on the python pyodbc library. You can find complete documentation about the library here.

There are a variety of ways to configure a pyodbc connection depending on your security implementation. A standard connection would consist of a host, port, database name, user, and password.

Fields

Field	Required	Default	Description
db-host	no		The MSSQL database host
db-port	no	1433	The MSSQL database port.
db-database-name	no		The database name
db-user	no	sa	The database user
db-password	no		The database password
encrypt	no		True/False Indicates whether to use an encrypted connection to mssql
trusted_connection	no		True/False whether the SQL Server connection is trusted. Sets `Trusted_Connection=yes` in pyodbc.
protocol	no	tcp	Connection protocol for the database. One of 'tcp', 'Icp', or 'NP'
server_connection_string	no		An optional ODBC server connection string to use when connecting to the server. These are usually constructed as `{protocol}:{host},{port}`. This
trust_server_certificate	no	true	Sets the ODBC connection string `TrustServerCertificate`

Example

on:
  - pull_request
name: SQL Server
jobs:
  test_mssql:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Grai Action
        uses: grai-io/grai-actions/mssql@master
        with:
          namespace: my_apps_grai_namespace
          api-key: my_grai_api_key
          action: tests
          source-name: prod-db
          grai-api-url: https://api.grai.io
          db-user: sa
          db-password: sa_password
          server_connection_string: tcp:myserver,1433
          trust_server_certificate: 'true'

Name		Name	Last commit message	Last commit date
Latest commit History 56 Commits
.github/workflows		.github/workflows
bigquery		bigquery
dbt		dbt
docs		docs
fivetran		fivetran
flat-file		flat-file
image		image
mssql		mssql
mysql		mysql
postgres		postgres
redshift		redshift
snowflake		snowflake
tests		tests
.actrc		.actrc
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Makefile		Makefile
README.md		README.md
base_readme.md		base_readme.md
document-action-inputs.py		document-action-inputs.py
event.json		event.json
requirements.txt		requirements.txt
run_test.sh		run_test.sh

grai-io/grai-actions

Folders and files

Latest commit

History

Repository files navigation

Grai Actions

Shared Fields

Authentication

Other Parameters

Notes and Caveats

Github Authentication

Explicit Workflow Permissions

Personal Access Tokens

Snowflake

Fields

Example

Redshift

Fields

Example

MySQL

Fields

Example

Flat File

Fields

Example

BigQuery

Fields

Example

Postgres

Fields

Example

dbt

Fields

Example

Fivetran

Fields

Namespace Map

Example

Microsoft SQL Server

Fields

Example

About

Resources

Stars

Watchers

Forks

Releases 1

Packages 0

Contributors 2

Languages

Packages