Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(snowflake-driver): Ability to use encrypted private keys for auth #9371

Merged
merged 5 commits into from
Mar 24, 2025
Merged

feat(snowflake-driver): Ability to use encrypted private keys for auth #9371

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d6fc834
feat(snowflake-driver): Ability to use encrypted private keys for auth
KSDaemon Mar 21, 2025
d47587d
update snowflake docs
KSDaemon Mar 21, 2025
f8a1f6d
add snowflake-encrypted-pk to ci drivers tests
KSDaemon Mar 21, 2025
28dd49f
fix
KSDaemon Mar 21, 2025
d7a5aee
add missing secrets
KSDaemon Mar 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/drivers-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ jobs:
redshift
redshift-export-bucket-s3
snowflake
snowflake-encrypted-pk
snowflake-export-bucket-s3
snowflake-export-bucket-azure
snowflake-export-bucket-azure-via-storage-integration
Expand Down Expand Up @@ -242,6 +243,7 @@ jobs:
- redshift
- redshift-export-bucket-s3
- snowflake
- snowflake-encrypted-pk
- snowflake-export-bucket-s3
- snowflake-export-bucket-azure
- snowflake-export-bucket-azure-via-storage-integration
Expand Down Expand Up @@ -340,6 +342,8 @@ jobs:
# Snowflake
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_USER: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_USER }}
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PASS: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PASS }}
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY }}
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS }}
with:
max_attempts: 3
retry_on: error
Expand Down
44 changes: 24 additions & 20 deletions docs/pages/product/configuration/data-sources/snowflake.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,16 @@ redirect_from:
- [The region][snowflake-docs-regions] for the [Snowflake][snowflake] warehouse
- The username/password for the [Snowflake][snowflake] account

## Snowflake quoted identifiers
## Snowflake quoted identifiers

Due to an issue in snowflakes opinion about quoted identifers we set a session value to override
Due to an issue in snowflakes opinion about quoted identifers we set a session value to override
snowflake defaults for users that have set an account value for: QUOTED_IDENTIFIERS_IGNORE_CASE
you can learn more about this here: https://docs.snowflake.com/en/sql-reference/identifiers-syntax#double-quoted-identifiers

## Setup

<WarningBox>
If you're having Network error and Snowflake can't be reached please make sure you tried
If you're having Network error and Snowflake can't be reached please make sure you tried
[format 2 for an account id][snowflake-format-2].
</WarningBox>

Expand Down Expand Up @@ -65,21 +65,25 @@ if [dedicated infrastructure][ref-dedicated-infra] is used. Check out the

## Environment Variables

| Environment Variable | Description | Possible Values | Required |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | :------: |
| `CUBEJS_DB_SNOWFLAKE_ACCOUNT` | The Snowflake account identifier to use when connecting to the database | [A valid Snowflake account ID][snowflake-docs-account-id] | ✅ |
| `CUBEJS_DB_SNOWFLAKE_REGION` | The Snowflake region to use when connecting to the database | [A valid Snowflake region][snowflake-docs-regions] | ❌ |
| `CUBEJS_DB_SNOWFLAKE_WAREHOUSE` | The Snowflake warehouse to use when connecting to the database | [A valid Snowflake warehouse][snowflake-docs-warehouse] in the account | ✅ |
| `CUBEJS_DB_SNOWFLAKE_ROLE` | The Snowflake role to use when connecting to the database | [A valid Snowflake role][snowflake-docs-roles] in the account | ❌ |
| `CUBEJS_DB_SNOWFLAKE_CLIENT_SESSION_KEEP_ALIVE` | If `true`, [keep the Snowflake connection alive indefinitely][snowflake-docs-connection-options] | `true`, `false` | ❌ |
| `CUBEJS_DB_NAME` | The name of the database to connect to | A valid database name | ✅ |
| `CUBEJS_DB_USER` | The username used to connect to the database | A valid database username | ✅ |
| `CUBEJS_DB_PASS` | The password used to connect to the database | A valid database password | ✅ |
| `CUBEJS_DB_SNOWFLAKE_AUTHENTICATOR` | The type of authenticator to use with Snowflake. Use `SNOWFLAKE` with username/password, or `SNOWFLAKE_JWT` with key pairs. Defaults to `SNOWFLAKE` | `SNOWFLAKE`, `SNOWFLAKE_JWT` | ❌ |
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PATH` | The path to the private RSA key folder | A valid path to the private RSA key | ❌ |
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS` | The password for the private RSA key. Only required for encrypted keys | A valid password for the encrypted private RSA key | ❌ |
| `CUBEJS_DB_MAX_POOL` | The maximum number of concurrent database connections to pool. Default is `20` | A valid number | ❌ |
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
| Environment Variable | Description | Possible Values | Required |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | :------: |
| `CUBEJS_DB_SNOWFLAKE_ACCOUNT` | The Snowflake account identifier to use when connecting to the database | [A valid Snowflake account ID][snowflake-docs-account-id] | ✅ |
| `CUBEJS_DB_SNOWFLAKE_REGION` | The Snowflake region to use when connecting to the database | [A valid Snowflake region][snowflake-docs-regions] | ❌ |
| `CUBEJS_DB_SNOWFLAKE_WAREHOUSE` | The Snowflake warehouse to use when connecting to the database | [A valid Snowflake warehouse][snowflake-docs-warehouse] in the account | ✅ |
| `CUBEJS_DB_SNOWFLAKE_ROLE` | The Snowflake role to use when connecting to the database | [A valid Snowflake role][snowflake-docs-roles] in the account | ❌ |
| `CUBEJS_DB_SNOWFLAKE_CLIENT_SESSION_KEEP_ALIVE` | If `true`, [keep the Snowflake connection alive indefinitely][snowflake-docs-connection-options] | `true`, `false` | ❌ |
| `CUBEJS_DB_NAME` | The name of the database to connect to | A valid database name | ✅ |
| `CUBEJS_DB_USER` | The username used to connect to the database | A valid database username | ✅ |
| `CUBEJS_DB_PASS` | The password used to connect to the database | A valid database password | ✅ |
| `CUBEJS_DB_SNOWFLAKE_AUTHENTICATOR` | The type of authenticator to use with Snowflake. Use `SNOWFLAKE` with username/password, or `SNOWFLAKE_JWT` with key pairs. Defaults to `SNOWFLAKE` | `SNOWFLAKE`, `SNOWFLAKE_JWT`, `OAUTH` | ❌ |
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY` | The content of the private RSA key | Content of the private RSA key (encrypted or not) | ❌ |
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PATH` | The path to the private RSA key | A valid path to the private RSA key | ❌ |
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS` | The password for the private RSA key. Only required for encrypted keys | A valid password for the encrypted private RSA key | ❌ |
| `CUBEJS_DB_SNOWFLAKE_OAUTH_TOKEN_PATH` | The path to the valid oauth toket file | A valid path for the oauth token file | ❌ |
| `CUBEJS_DB_SNOWFLAKE_HOST` | Host address to which the driver should connect | A valid hostname | ❌ |
| `CUBEJS_DB_SNOWFLAKE_QUOTED_IDENTIFIERS_IGNORE_CASE` | Whether or not quoted identifiers should be case insensitive. Default is `false` | `true`, `false` | ❌ |
| `CUBEJS_DB_MAX_POOL` | The maximum number of concurrent database connections to pool. Default is `20` | A valid number | ❌ |
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |

[ref-data-source-concurrency]: /product/configuration/concurrency#data-source-concurrency

Expand Down Expand Up @@ -160,8 +164,8 @@ CUBEJS_DB_EXPORT_INTEGRATION=gcs_int

#### Azure

To use Azure Blob Storage as an export bucket, follow [the guide on
using a Snowflake storage integration (Option 1)][snowflake-docs-azure].
To use Azure Blob Storage as an export bucket, follow [the guide on
using a Snowflake storage integration (Option 1)][snowflake-docs-azure].
Take note of the integration name (`azure_int` from the example link)
as you'll need it to configure Cube.

Expand Down
29 changes: 26 additions & 3 deletions packages/cubejs-snowflake-driver/src/SnowflakeDriver.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import {
} from '@cubejs-backend/base-driver';
import { formatToTimeZone } from 'date-fns-timezone';
import fs from 'fs/promises';
import crypto from 'crypto';
import { HydrationMap, HydrationStream } from './HydrationStream';

const SUPPORTED_BUCKET_TYPES = ['s3', 'gcs', 'azure'];
Expand Down Expand Up @@ -245,8 +246,30 @@ export class SnowflakeDriver extends BaseDriver implements DriverInterface {
assertDataSource('default');

let privateKey = getEnv('snowflakePrivateKey', { dataSource });
if (privateKey && !privateKey.endsWith('\n')) {
privateKey += '\n';

if (privateKey) {
// If the private key is encrypted - we need to decrypt it before passing to
// snowflake sdk.
if (privateKey.includes('BEGIN ENCRYPTED PRIVATE KEY')) {
const keyPasswd = getEnv('snowflakePrivateKeyPass', { dataSource });

if (!keyPasswd) {
throw new Error(
'Snowflake encrypted private key provided, but no passphrase was given.'
);
}

const privateKeyObject = crypto.createPrivateKey({
key: privateKey,
format: 'pem',
passphrase: keyPasswd
});

privateKey = privateKeyObject.export({
format: 'pem',
type: 'pkcs8'
});
}
}

snowflake.configure({ logLevel: 'OFF' });
Expand All @@ -266,7 +289,7 @@ export class SnowflakeDriver extends BaseDriver implements DriverInterface {
oauthTokenPath: getEnv('snowflakeOAuthTokenPath', { dataSource }),
privateKeyPath: getEnv('snowflakePrivateKeyPath', { dataSource }),
privateKeyPass: getEnv('snowflakePrivateKeyPass', { dataSource }),
privateKey,
...(privateKey ? { privateKey } : {}),
exportBucket: this.getExportBucket(dataSource),
resultPrefetch: 1,
executionTimeout: getEnv('dbQueryTimeout', { dataSource }),
Expand Down
9 changes: 9 additions & 0 deletions packages/cubejs-testing-drivers/fixtures/snowflake.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,15 @@
"CUBEJS_DB_EXPORT_GCS_CREDENTIALS": "${DRIVERS_TESTS_CUBEJS_DB_EXPORT_GCS_CREDENTIALS}"
}
}
},
"encrypted-pk": {
"cube": {
"environment": {
"CUBEJS_DB_SNOWFLAKE_AUTHENTICATOR": "SNOWFLAKE_JWT",
"CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY": "${DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY}",
"CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS": "${DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS}"
}
}
}
},
"cube": {
Expand Down
1 change: 1 addition & 0 deletions packages/cubejs-testing-drivers/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
"snowflake-driver": "yarn test-driver -i dist/test/snowflake-driver.test.js",
"snowflake-core": "yarn test-driver -i dist/test/snowflake-core.test.js",
"snowflake-full": "yarn test-driver -i dist/test/snowflake-full.test.js",
"snowflake-encrypted-pk-full": "yarn test-driver -i dist/test/snowflake-encrypted-pk-full.test.js",
"snowflake-export-bucket-s3-full": "yarn test-driver -i dist/test/snowflake-export-bucket-s3-full.test.js",
"snowflake-export-bucket-azure-full": "yarn test-driver -i dist/test/snowflake-export-bucket-azure-full.test.js",
"snowflake-export-bucket-azure-via-storage-integration-full": "yarn test-driver -i dist/test/snowflake-export-bucket-azure-via-storage-integration-full.test.js",
Expand Down
Loading
Loading