Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Server DNS - The message or signature supplied for verification has been altered #215

Open
1 task done
storkeyp opened this issue Jun 15, 2022 · 12 comments
Open
1 task done
Labels

Comments

@storkeyp
Copy link

storkeyp commented Jun 15, 2022

Terraform CLI and Provider Versions

Terraform v1.2.1
on windows_amd64

  • provider registry.terraform.io/hashicorp/dns v3.2.3
  • provider registry.terraform.io/hashicorp/null v3.1.1
  • provider registry.terraform.io/hashicorp/vault v3.6.0
  • provider registry.terraform.io/hashicorp/vsphere v2.1.1
  • provider registry.terraform.io/ouest-france/phpipam v0.6.0

Terraform Configuration

provider "dns" {
    update {
      server                  = "${var.domain_controller}.${var.domain_name}"
      gssapi {
        realm                 = upper("${var.domain_name}")
        username              = var.username
        password              = var.password
      }
    }
}

resource "dns_a_record_set" "wyse" {
    zone = "${var.dns_cname}.${var.dns_zone}"
    addresses = [
        phpipam_address.wms_ip_address[0].ip_address
    ]
}

Expected Behavior

DNS A record added to zone, or modified if it already exists

Actual Behavior

A record is successfully created, but Terraform apply fails with the following error:

Error: Error updating DNS record: The message or signature supplied for verification has been altered

with module.vms.dns_a_record_set.wyse,
on ..\modules\vms\wms.tf line 99, in resource "dns_a_record_set" "wyse":
99: resource "dns_a_record_set" "wyse" {

Steps to Reproduce

  1. terraform apply

How much impact is this issue causing?

Medium

Logs

No response

Additional Information

Windows Server 2012 R2 Standard, domain integrated zone

Code of Conduct

  • I agree to follow this project's Code of Conduct
@storkeyp storkeyp added the bug label Jun 15, 2022
@SpeedHighway
Copy link

SpeedHighway commented Jun 20, 2022

This one's occuring for us, as well, but we're using a cname and doing an edit, for an imported record. Unfortunately, the edit did NOT occur, in our case.

EDIT: For clarification, we're trying to add/edit entries to 3 different DNS servers, and getting this result on all 3.

EDIT2: Is it possible that the RR messages are being created incorrectly? I find it hard to believe it wouldn't have been noticed before now if this is a problem, but all of the test cases inside the dns project's repository use the phrases "IN A" or "IN CNAME", whereas, this project is setting them up as "A" or "CNAME", without the word "IN". I don't know if it makes a difference, but wonder if it could. https://github.com/miekg/dns/blob/40060b4a4b85b14867003343f64bfd5cd64ea256/duplicate_test.go

@mtlucas
Copy link

mtlucas commented Aug 2, 2022

I had the same problem and fixed it by changing my DNS zone Properties to "Dynamic updates: Secure only". It was failing when I had "Nonsecure and Secure" selected.

EDIT: OK, this happened again even after changing to secure updates. Seems intermittent and still problematic.

@RahmanBadru
Copy link

Has anybody been able to solve this, i have the same issue and hence i cant destroy the records and achieve desired state

@ranExl
Copy link

ranExl commented Nov 17, 2022

Same here. Did anyone manage to pass this one?

@Aiikon
Copy link

Aiikon commented Nov 18, 2022

Try making your DNS server name all lowercase if it's not.

I had the exact same issue. I finally tested on a RHEL server and got a similar sounding error mentioned in #213 and the fix was the same. Changing server to lowercase worked for both Windows and Linux.

@ranExl
Copy link

ranExl commented Nov 20, 2022

Getting a different error now -
Error: Error updating DNS record: read udp <client_ip>:65428-><server_ip>:53: i/o timeout

@Aiikon
Copy link

Aiikon commented May 15, 2023

I've found this error also occurs with AD/GSSAPI on the Windows when multiple records are being updated simultaneously. It seems somewhere in the pipeline the DNS module is not thread safe. I've been able to work around the issue forcing a single thread using "terraform apply -parallelism 1" but this is only usable if DNS resources aren't paired with resources that take a long time to provision like VMs. I haven't had a chance to test if parallelism is a problem or not on Linux. I wish there was a configuration to control the thread count per-resource or provider to make this easier.

@jayo86
Copy link

jayo86 commented Jul 22, 2023

Try making your DNS server name all lowercase if it's not.

I had the exact same issue. I finally tested on a RHEL server and got a similar sounding error mentioned in #213 and the fix was the same. Changing server to lowercase worked for both Windows and Linux.

fyi this fixed it for me

@cbus-guy
Copy link

We have the same issue. Don't know why this isn't getting any traction. ???

@bendbennett
Copy link
Contributor

Hi @storkeyp 👋

Sorry you ran into trouble here. Can I confirm the following:

  • The issue you reported is still present when using the latest version of the tls provider, currently v3.4.0, and if possible the latest version of Terraform, currently v1.7.5.
  • The issue persist when the server is specified in lowercase, as described by @Aiikon.

If the issue persists, then it would be very helpful if you could gather the details necessary to reproduce this issue in a local development environment. Thanks.

@rahmnstein
Copy link

rahmnstein commented Nov 18, 2024

Just installed a new test domain controller.
OS: 2022
Server FQDN: dc1.mylab.corp
Forest / Domain: mylab.corp

Ran the code on the domain controller itself.

Four "terraform apply" below shows the error.

  1. It succeed with 2 / 4 records.
  2. It succeed with remaining 2 records.
  3. Success, can refresh all four records! "No changes. Your infrastructure matches the configuration."
  4. Fail to refresh all four records.

If I change 'server = "dc1.mylab.corp"' to server = "mylab.corp" I get error 'The message or signature supplied for verification has been altered' instead.

It is like it logs in once for every record at the same time and the DC says no. If I only create one DNS-record and then remove it again, it never fails. If I have more than one record in the state, it need to refresh all of them at the same time and it fails sometimes. Not every time but often.

Do you have a lab where you actually can add multiple DNS-records in the same apply on a AD domain controller?

terraform {
  required_providers {
    dns = {
      source  = "hashicorp/dns"
      version = "3.4.2"
    }
  }
}


provider "dns" {
  update {
    server = "dc1.mylab.corp"
    gssapi {
      realm    = "MYLAB.CORP"
      username = "administrator"
      password = "AmazingPw123!!"
    }
  }
}
resource "dns_a_record_set" "records" {
  count = 4

  zone = "mylab.corp."
  name = "testing${count.index + 1}"
  addresses = [
    "1.2.3.4",
  ]
  ttl = 300
}
C:\tf>.\terraform.exe apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # dns_a_record_set.records[0] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing1"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[1] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing2"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[2] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing3"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[3] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing4"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

dns_a_record_set.records[3]: Creating...
dns_a_record_set.records[0]: Creating...
dns_a_record_set.records[1]: Creating...
dns_a_record_set.records[2]: Creating...
dns_a_record_set.records[3]: Creation complete after 0s [id=testing4.mylab.corp.]
dns_a_record_set.records[2]: Creation complete after 0s [id=testing3.mylab.corp.]
╷
│ Error: Error updating DNS record: dns: no secrets defined
│
│   with dns_a_record_set.records[0],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵
╷
│ Error: Error updating DNS record: dns: no secrets defined
│
│   with dns_a_record_set.records[1],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵

C:\tf>.\terraform.exe apply
dns_a_record_set.records[3]: Refreshing state... [id=testing4.mylab.corp.]
dns_a_record_set.records[2]: Refreshing state... [id=testing3.mylab.corp.]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # dns_a_record_set.records[0] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing1"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[1] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing2"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

dns_a_record_set.records[0]: Creating...
dns_a_record_set.records[1]: Creating...
dns_a_record_set.records[0]: Creation complete after 0s [id=testing1.mylab.corp.]
dns_a_record_set.records[1]: Creation complete after 0s [id=testing2.mylab.corp.]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

C:\tf>.\terraform.exe apply
dns_a_record_set.records[2]: Refreshing state... [id=testing3.mylab.corp.]
dns_a_record_set.records[0]: Refreshing state... [id=testing1.mylab.corp.]
dns_a_record_set.records[3]: Refreshing state... [id=testing4.mylab.corp.]
dns_a_record_set.records[1]: Refreshing state... [id=testing2.mylab.corp.]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are
needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

C:\tf>.\terraform.exe apply
dns_a_record_set.records[1]: Refreshing state... [id=testing2.mylab.corp.]
dns_a_record_set.records[0]: Refreshing state... [id=testing1.mylab.corp.]
dns_a_record_set.records[3]: Refreshing state... [id=testing4.mylab.corp.]
dns_a_record_set.records[2]: Refreshing state... [id=testing3.mylab.corp.]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Error querying DNS record: error negotiating GSS context: 1 error occurred:
│       * DNS error: REFUSED (5)
│
│
│
│   with dns_a_record_set.records[0],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵
╷
│ Error: Error querying DNS record: dns: no secrets defined
│
│   with dns_a_record_set.records[3],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵

C:\tf>.\terraform.exe version
Terraform v1.9.8
on windows_amd64
+ provider registry.terraform.io/hashicorp/dns v3.4.2

@Aiikon
Copy link

Aiikon commented Nov 19, 2024

I see that parallel CRUDs are still an issue with the latest version of this provider. Hopefully this explanation and example helps some people or gets some traction towards fixing the issue. I have a customized provider that I use internally at work that you can use if you're willing to build your own copy of the provider so scroll down if you want to try that.

First make sure you are able to update a single record reliably. A few issues that may result in cryptic errors are:

  • Using an IP address or incorrect case for the server name (it seems to want lowercase).
  • A bad or missing password.

The DNS error: REFUSED errors seem to be because some component of the DNS or kerberos library has threading issues. This can be tested by running terraform plan/apply/destroy with the --parallelism 1 option. Here is a sample configuration that will fail in a lab environment every time unless ran with terraform apply --parallelism 1 (at which point it succeeds 100% of the time):

terraform {
  required_providers {
    dns = {
      source = "hashicorp/dns"
      version = "3.4.2"
    }
  }
}

provider "dns" {
  update {
  server = "win-qbs0pliia8g"
    gssapi {
      realm = "domain1.local"
      username = "Administrator"
      password = "------------"
    }
  }
}

locals {
  records = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"]
}

resource "dns_a_record_set" records {
  for_each = toset(local.records)

  zone = "domain1.local."
  name = "server-${each.key}"
  addresses = ["10.0.0.99"]
}

If you're only managing DNS or other instantaneous API calls you may be happy running everything with the parallelism flag. It'll still be pretty fast. If you're running other operations like VM provisioning though you probably want everything else running in parallel, so until the provider is fixed you can try my workaround.

Adding a few mutex lock/unlock lines to the resource files and building your own provider is really easy. It's a hack, probably the wrong way to do it, and messy but I use an altered provider in a large VM build configuration and it works for both windows and linux clients (and TFE if you publish it internally). Terraform isn't aware only one resource can be accessed at a time so the logs may show some updates taking longer than they really do.

To customize the provider so only one CRUD operation can be performed at a time:

provider.go needs a global variable added near the top:

var mutex sync.Mutex

and each resource_*.go file needs a mutex lock and unlock added at the start of each CRUD function:

func (d *dnsCNAMERecordResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {

mutex.Lock()
defer mutex.Unlock()

The A record's read function is also used by update so it needs another definition that doesn't have a mutex or the provider will wait forever, it's harder to explain so check the fork I linked.

Here's a fork containing updates to the A and CNAME resource files. I've only tested these resources so you would need to add the mutex to other resources like AAAA or SRV yourself :

Building the provider is the easy part. Just install go and build with go build and it should complete nearly instantly (at least it does for me with vscode). Getting terraform to use it can be trickier. Some of your options are:

  • Add it to your own provider mirror if you use one: https://developer.hashicorp.com/terraform/internals/provider-network-mirror-protocol. I think I remember you can skip the signing/hashes with this.
  • Sign and publish it in your own TFE/TFCloud instance (what I have setup at work).
  • Use terraform init -plugin-dir which is a decent quick test that I just did in my lab.
    • Do a regular terraform init with the normal DNS module, then delete .terraform.lock.hcl and rename the .terraform directory to plugins, replace the hashicorp terraform-provider-dns_v3.4.2_x5.exe in plugins with your newly built one, then re-run terraform init -plugin-dir "my-workspace-path\plugins\providers". Change the provider name to a custom one if you make this permanent. The terraform plan/apply/destroy commands should now work without needing to limit parallelism for the entire project.

I don't want this to become a troubleshooting thread for using in-house providers so I'll leave it at that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

10 participants