Skip to content

4.2.0
Compare
Choose a tag to compare
@github-actions github-actions released this 19 Jul 12:54
· 191 commits to master since this release
4.2.0
911e2c9

New authorization options for Snowflake and Databricks

Up until version 4.1.0, RDB Loader required that the warehouse had been pre-configured to have read-access to the data files in S3.
For Snowflake, this meant setting up an external stage with a storage integration.
For Databricks, it meant setting up a cluster to assume an AWS instance profile.

Starting with version 4.2.0, RDB Loader is able to generate temporary credentials using STS and pass these credentials to Snowflake/Databricks. This removes the need to pre-configure the warehouse with access permission.

To start using the new authorization method, you must add a loadAuthMethod to the storage block in your config file:

"storage": {
  // other required fields go here
  "loadAuthMethod": {
    "type": "TempCreds"
    "roleArn": "arn:aws:iam::123456789:role/example_role_name"
  }
}

...where roleArn is a role with permission to read files from the S3 bucket. The loader must have permission to assume this role.
Our Github repo has some examples of this configuration for Snowflake and for Databricks.

Note, for Snowflake loading, depending on your event volume and warehouse configuration, there may still be an advantage to setting up the storage integration, because the underlying COPY INTO statement is more efficient.
For Databricks loading, though, there should be no impact of changing to the new authorization method.

Retry on target initialization

Initialization block is surrounded by retry block so that if an exception is thrown from initialization block instead of crashing the application, it will be retried according to the specified backoff strategy.

To enable this feature, initRetries must be added to config file:

  "initRetries": {
    "backoff": "30 seconds"
    "strategy": "EXPONENTIAL"
    "attempts": 3,
    "cumulativeBound": "1 hour"
  },

Adjusting the path appended to Snowflake stage in Snowflake Loader

As it is mentioned in the first section, we've added a new authorization option to Snowflake Loader. However, old Snowflake stage method still can be used.

Previously, Snowflake stage path needs to be exactly the path where transformed run folders reside. If the path of an upper folder is given as a stage path, loading wouldn't work.

We've fixed this issue in this release. Even if the stage path is set to the path of transformed folder's upper directory, loading would still work correctly.

To use this feature, you need to update transformedStage and folderMonitoringStage blocks:

  "transformedStage":  {
    # The name of the stage
    "name": "snowplow_stage"
    # The S3 path used as stage location
    "location": "s3://bucket/transformed/"
  }
  "folderMonitoringStage": {
    # The name of the stage
    "name": "snowplow_folders_stage"
    # The S3 path used as stage location
    "location": "s3://bucket/monitoring/"
  }

Bug fix for streaming transformer on multiple instances

In the previous released of RDB Loader we announced that the streaming transformer can now scale to multiple instances, which was a really important requirement for high volume pipelines.
We got one little thing wrong though, and it lead to some app crashes with error messages about lost Kinesis leases. This bug is now fixed in version 4.2.0, and we hope this unlocks your pipeline from scaling to higher event volumes with the streaming transformer.

Upgrading to 4.2.0

If you are already using a recent version of RDB Loader (3.0.0 or higher) then upgrading to 4.2.0 is as simple as pulling the newest docker images. There are no changes needed to your configuration files.

docker pull snowplow/transformer-kinesis:4.2.0
docker pull snowplow/rdb-loader-redshift:4.2.0
docker pull snowplow/rdb-loader-snowflake:4.2.0
docker pull snowplow/rdb-loader-databricks:4.2.0

The Snowplow docs site has a full guide to running the RDB Loader.

Changelog

  • Transformer kinesis: Recover from losing lease to a new worker (#962)
  • Loader: make the part appended to folder monitoring staging path configurable (#969)
  • Loader: retry on target initialization (#964)
  • Loader: Trim alert message payloads to 4096 characters (#956)
  • Snowflake loader: make on_error continue when type of the incoming data is parquet (#970)
  • Snowflake loader: make the path used with stage adjustable (#968)
  • Snowflake loader: use STS tokens for copying from S3 (#955)
  • Snowflake loader: Specify file format in the load statement (#957)
  • Databricks loader: Generate STS tokens for copying from S3 (#954)
Assets 7
Loading