Is it possible to support more events for workspace hooks ?

[BenjaminDecreusefond]

Hi !

I've been looking to the possibility to implement the trigger of speculative plans when a PR is opened. I've been checking at the provider code and I see an interesting resource here. The issue is that it triggers only when a PUSH is performed on the branch. My idea is that we could add a new trigger event, for instance Open PR and trigger the webhook following that event.

I wonder if this seem doable to you or if there are any blocking points ? If you say it's possible with the current state of the API i could take a look at it.

Regards !

[alfespa17]

It will allow you to create more web hooks but it only works for push events

https://github.com/AzBuilder/terrakube/blob/e30c9863d034fb8b7908ce566dab2a08a276215d/api/src/main/java/org/terrakube/api/plugin/vcs/provider/github/GitHubWebhookService.java#L58

[BenjaminDecreusefond]

Do you think it would be technically feasible to add a handler for PR and open PR for instance ?

[alfespa17]

Do you think it would be technically feasible to add a handler for PR and open PR for instance ?

It will require some work and research for that type of event maye in the future that feature can be added

[BenjaminDecreusefond]

I'd like to help as our company really need this but I uncomfortable to test on Terrakube repo. Building images using maven is a bit complicated :/

[BenjaminDecreusefond]

Or maybe a workaround is to use the workspace webhook with PUSH on any branch ? So that when a PR is updated AKA a push is made on the branch of the PR, it triggers a run ?

[alfespa17]

I'd like to help as our company really need this but I uncomfortable to test on Terrakube repo. Building images using maven is a bit complicated :/

Not sure why you are saying building images is complicated it only require one command "mvn clean intall" using maven and java jdk 17, you can fork the repo and try using this if you want to test development in some point and you dont want to do any setup in your machine

[BenjaminDecreusefond]

Do you know if the branch parameter of the workspace_webhook of the terrakube provider is working with regex ? For instance setting the branch as * would trigger for any branch ?

[alfespa17]

Do you know if the branch parameter of the workspace_webhook of the terrakube provider is working with regex ? For instance setting the branch as * would trigger for any branch ?

You can use it like this, and it will check if the name of the branch starts with those values

Here is the method that validates the branch name

https://github.com/AzBuilder/terrakube/blob/e30c9863d034fb8b7908ce566dab2a08a276215d/api/src/main/java/org/terrakube/api/plugin/vcs/WebhookService.java#L209

[BenjaminDecreusefond]

Thanks I'll try to mix up a solution with that !

[stanleyz]

hmm, interesting, I am also thinking of enhancing the PR-based runs and happened to see this.

I did some reading previously, and we will need to register pull-request events, I think Opened, reopened and synchronize are the ones we should kick off runs against. I am happy to help @BenjaminDecreusefond if you have some draft PR already.

[BenjaminDecreusefond]

Hey @stanleyz !

Adding new type of events seemed to be a bit tricky to implement to me, so instead I think I can achieve what I need by enhancing the branch matching to support wildcards * and be able to exclude some branches. For instance ["*", "!master"] would trigger the work for all branches except master. This can be useful as we want to run speculative plans on branch that are not the default branch and run apply when we merge. Btw this brings me to two questions that I've got. I've been trying to add webhooks to a workspace and I got an issue while trying to create it with the resource.

  # terrakube_workspace_webhook.webhook will be created
  + resource "terrakube_workspace_webhook" "webhook" {
      + branch          = [
          + "feat",
          + "fix",
        ]
      + event           = "PUSH"
      + id              = (known after apply)
      + organization_id = "179acb9d-5119-49db-a7b6-ed3aeca88daa"
      + path            = [
          + "/projects/my_test/.*.tf",
        ]
      + remote_hook_id  = (known after apply)
      + template_id     = "9163e61b-b12e-4f72-a3f5-5eb5918baaa6"
      + workspace_id    = "5b58f17d-d440-4a4e-af40-bcdf895cd1ca"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

terrakube_workspace_webhook.webhook: Creating...
╷
│ Error: Error unmarshal payload response
│ 
│   with terrakube_workspace_webhook.webhook,
│   on workspace.tf line 21, in resource "terrakube_workspace_webhook" "webhook":
│   21: resource "terrakube_workspace_webhook" "webhook" {
│ 
│ Error unmarshal payload response: response status 424 Failed Dependency, response body: {%!s(*http.http2clientStream=&{0x140001fe300 0x14000114010 <nil> <nil> 1
│ {{0 0} {{} 0x140001fe4b0 {0 0 0 <nil> <nil>} 1374391624936} <nil> 0 0x1400010e110 <nil> <nil> <nil>} true false {0 {0 0}} 0x14000043080 <nil> 0x14000043020
│ 0x14000043140 <nil> 0x140000430e0 0x14000548e10 {[] 65307 0x140001fe360} {[] 4194224 0x140001fe370} 0 <nil> 0x140003b94d0 229 true true true true true false 0 true
│ false map[] 0x14000548e88})}, error: data is not a jsonapi representation of '*client.WorkspaceWebhookEntity'
╵

Also, I'm wondering if it is possible to attach several terrakube_workspace_webhook to a single workspace ? This way we could attach one webhook to trigger plans when not pushing on master and add another webhook that trigger the apply template when we merge on default branch. ?

Thanks !

[BenjaminDecreusefond]

Update on the issue !

Actually I had this error because I was trying to have the webhook to a CLI workspace !

Secondly, It is possible to have several webhooks for a single workspace however, the UI will only display one of them ! Maybe there is something to do about this ?

[stanleyz]

Actually I had this error because I was trying to have the webhook to a CLI workspace !

Yes, it has to be VCS workspace and the VCS app need webhook permission to register the webhook settings in the repository.

Secondly, It is possible to have several webhooks for a single workspace however, the UI will only display one of them ! Maybe there is something to do about this ?

Currently only PUSH events are listened in Terrakube, if you are not actively working on PR event, I'll take a look in the next one or two weeks depending on my workload :)

[BenjaminDecreusefond]

Hey @stanleyz !

I just fired up this PR ! Lemme know if you think that would do the trick !

Hopefully I'd like to add this PR and merge the other one to enhance branch matching so I can achieve our goal of speculative plans on PRs !

[stanleyz]

This way we could attach one webhook to trigger plans when not pushing on master and add another webhook that trigger the apply template when we merge on default branch. ?

You can achieve this with the new 2.24.0-beta version by setting PLAN template for webhook and PLAN/APPLY template on the workspace default branch, related code below:

https://github.com/AzBuilder/terrakube/blob/ea1103c93f722585dce74c56907cfd520bc50005/api/src/main/java/org/terrakube/api/plugin/vcs/WebhookService.java#L94C2-L107C1

[BenjaminDecreusefond]

Yep I figured it out yesterday while testing ahah, thanks !

[BenjaminDecreusefond]

@stanleyz Do you know if there could be a way to retrieve on PRs the run that has been triggered on Terrakube and display a link to the job on the current PR ?

[stanleyz]

@stanleyz Do you know if there could be a way to retrieve on PRs the run that has been triggered on Terrakube and display a link to the job on the current PR ?

Yes, I have added some information to your PR, you need to update the sendCommitStatus to do so, you should be able to find the commit ID from the payload of the PR event.

https://github.com/AzBuilder/terrakube/blob/1eb779e5b0fb6aae3b5a148f5bea7aec9d7900eb/api/src/main/java/org/terrakube/api/plugin/vcs/provider/github/GitHubWebhookService.java#L154

[BenjaminDecreusefond]

thanks @stanleyz I updated the PR in consequence ! :)

[alfespa17]

Update on the issue !

Actually I had this error because I was trying to have the webhook to a CLI workspace !

Secondly, It is possible to have several webhooks for a single workspace however, the UI will only display one of them ! Maybe there is something to do about this ?

There is one thing that you need to keep in mind, there is a limit restriction in github you can have up to 20 webhooks inside a repository and Terrakube creates one webhook for every workspace so in theory you can have up to 20 workspaces inside a git repo

By the way I never tested adding multiple webhooks for a single workspace so I have no idea if that should work

There is an open issue about that to have only 1 webhook:

AzBuilder/terrakube#672

The above issue will need a really big refactor in the logic

[BenjaminDecreusefond]

Hi @alfespa17 !

Thanks for you return !

I tried to create severals webhooks and it did actually works but in a weird way ! The UI was only displaying one webhook but on github there was 2 webhooks created !
The limitation is kind of a blocking point as we plan to have way more than 20 workspaces using VCS :/
How big the refactor is according to you ? It could be quite a breaking point :/
I'm thinking, maybe a quick win would to be able to attach one webhook to severals workspaces ?
Using this resource and convert the workspace_id to contain multiple workspace ids and handle the logic in the API, would that imply many change ?

[alfespa17]

Hi @alfespa17 !

Thanks for you return !

I tried to create severals webhooks and it did actually works but in a weird way ! The UI was only displaying one webhook but on github there was 2 webhooks created ! The limitation is kind of a blocking point as we plan to have way more than 20 workspaces using VCS :/ How big the refactor is according to you ? It could be quite a breaking point :/ I'm thinking, maybe a quick win would to be able to attach one webhook to severals workspaces ? Using this resource and convert the workspace_id to contain multiple workspace ids and handle the logic in the API, would that imply many change ?

It is a really good refactor and it will require a lot of time to change the logic in several places while keeping the code compatible with previous releases it is not a simple change.

I'm thinking, maybe a quick win would to be able to attach one webhook to severals workspaces ? is not that simple .....

[BenjaminDecreusefond]

Yep i didn't mean we simply need tochange the workspace_id to workspace_ids ahah 😂 It was just an idea to start the implementation. I do agree that without that feature it is difficult to use the VCS feature in production environment as we're very limited in workspace number :/

[BenjaminDecreusefond]

Is it planned to be changed in future release ?

[alfespa17]

Is it planned to be changed in future release ?

In the future it will change but there is no target date for that