terraform-provider-confluent-schema-registry

A terraform provider for managing schemas in a Confluent schema registry

Developing at Fetch

The local development workflow hashicorp/terraform-provider-aws#5396 08:32

1. make go code change
2. make build
3. cp to new place in plugins dir 4. $ cp dist/terraform-provider-schemaregistry ~/.terraform.d/plugins/local/fetch-rewards/confluent-schema-registry/1.1.0/darwin_arm64/terraform-provider-confluent-schema-registry_1.1.0
4. delete providers in .terraform
5. delete .terraform.lock.hcl
6. terraform init

Provider configuration

terraform {
    required_providers {
        schemaregistry = {
            source = "fetch-rewards/confluent-schema-registry"
        }
    }
}

provider "schemaregistry" {
    schema_registry_url = "https://<env>-event-tracking-schema-registry.fetchrewards.com"
    # Below configs are not needed for our SR as we dont have auth enabled
    # username            = "<confluent_schema_registry_key>"
    # password            = "<confluent_schema_registry_password>"
}

You can omit the credential details by defining the environment variables SCHEMA_REGISTRY_URL, SCHEMA_REGISTRY_USERNAME, SCHEMA_REGISTRY_PASSWORD

The schema resource

resource "schemaregistry_schema" "main" {
  subject = "<subject_name>"
  schema  = file("<avro_schema_file>")
}

The schema resource with references

Schema registry references can be used to allow putting Several Event Types in the Same Topic. Please refer to Confluent Schema Registry API Reference for details.

Upgrade reference version to the latest event schema version

Reference the event schema resource from a schema with reference wil upgrade a reference alongside with its referenced schema.

resource "schemaregistry_schema" "referenced_event" {
  subject = "referenced_event_subject"
  schema  = "{\"type\":\"record\",\"name\":\"event\",\"namespace\":\"akc.test\",\"fields\":[{\"name\":\"foo\",\"type\":\"string\"}]}"
}

resource "schemaregistry_schema" "other_referenced_event" {
  subject = "other_referenced_event_subject"
  schema  = "{\"type\":\"record\",\"name\":\"other_event\",\"namespace\":\"akc.test\",\"fields\":[{\"name\":\"bar\",\"type\":\"string\"}]}"
}

resource "schemaregistry_schema" "with_reference" {
  subject = "with_reference_subject"
  schema = "[\"akc.test.event\", \"akc.test.other_event\"]"

  reference {
    name = "akc.test.event"
    subject = schemaregistry_schema.referenced_event.subject
    // version will always be upgraded with the referenced event schema version  
    version = schemaregistry_schema.referenced_event.version
  }

  reference {
    name = "akc.test.other_event"
    subject = schemaregistry_schema.user_added.subject
    // version will always be upgraded with the referenced event schema version  
    version = schemaregistry_schema.referenced_event.version
  }
}

Stick reference version to a given version

Use a dataSource to stick a reference to a given version, while upgrading the referenced event schema.

resource "schemaregistry_schema" "referenced_event_latest" {
  subject = "referenced_event_subject"
  schema  = file("<avro_schema_file_updated>")
}

data "schemaregistry_schema" "referenced_event_v1" {
  subject = "other_referenced_event_subject"
  schema  = "{\"type\":\"record\",\"name\":\"other_event\",\"namespace\":\"akc.test\",\"fields\":[{\"name\":\"bar\",\"type\":\"string\"}]}"
}

resource "schemaregistry_schema" "with_reference_to_v1" {
  subject = "with_reference_subject"
  schema = "[\"akc.test.event\", \"akc.test.other_event\"]"

  references {
    name = "akc.test.event"
    subject = data.schemaregistry_schema.referenced_event_v1.subject
    version = data.schemaregistry_schema.referenced_event_v1.version
  }
}

The schema data source

data "schemaregistry_schema" "main" {
  subject = "<subject_name>"
}

output "schema_id" {
  value = data.schemaregistry_schema.main.id
}

output "schema_version" {
  value = data.schemaregistry_schema.main.version
}

output "schema_string" {
  value = data.schemaregistry_schema.main.schema
}

Importing an existing schema

terraform import schemaregistry_schema.main <subject_name>

Local testing/development

The test-new-build.sh script can be used to easily test the provider locally. It will build the provider, copy it to the local plugins directory, and run terraform init in the terraform-test-files directory.

From the terraform-test-files directory, you can run your commands as you normally would to test your provider changes, ie terraform apply, terraform plan, etc.

To see any logging that you added to the provider (using tflog) you can add the environment variable TF_LOG=INFO to your command, ie TF_LOG=INFO terraform apply.

An example of using tflog to log a message in the provider is below:

tflog.Info(ctx, "Configuring SchemaRegistry client")