Jobs should allow to set branch

[juan-vg]

Feature description 💡

When creating a new Job for a workspace, apart from templateReference and the workspace ID, it would be so useful to be able to set also the VCS branch to run the job on. This will allow PR speculative plans (and many other possibilities, I believe) to ensure the code that will ―eventually― end up in master branch, is working properly.

This branch provided to the Job must only be used for that job. I mean, the default branch configured for the workspace must not be changed. This will allow several runs (plans) on different branches in parallel, while still allowing the default branch to be planned and applied. This implies that applies must only run on the default branch, and also that if a different branch is provided, apply should not be an option (or the other way round: branch can only be provided if plan only).

This must be implemented for both UI and API

Anything else?

I believe the API docs for the jobs endpoint is not properly documented (at least the POST method) regarding the body attributes. As an example, templateReference is only mentioned at the top-level example.

[alfespa17]

There is one filed to override the branch in a Job but it is only used internally (to be honest I don't remember why we exclude it from the API), we use that field a lot when using the CLI driven workflow, we also use it with the VCS webhooks because you can have like multiple branches

terrakube/api/src/main/java/org/terrakube/api/rs/job/Job.java

Line 79 in b092fe7

private String overrideBranch;

The API docs need to be update if you want to get the latest spec you can do a GET to [terrakube-api.mydomain.com/doc](http://terrakube-api.mydomain.com/doc) that will return the json file and you can use https://editor.swagger.io/ to see it

[alfespa17]

For speculative plan at least when using github and using the workspace webhook you can define the branch name using a regex expression like ^feat.*,^fix.* (this will trigger any branch that name starts with feat or fix

The above will trigger a plan for any with feat or fix and when using the main branch you could put "Plan and Apply" like

And you could see the checks in github when creating a PR

There was a discussion about that here

https://terrakubeworkspace.slack.com/archives/C06JW2A4MH9/p1733411772269669

[juan-vg]

Regarding webhooks:

• It's not viable for my case since my TerraKube instance is private, so Github is not able to reach it. That's why I wanted to use the GitHub Actions running on self-hosted runners that are able to reach the TK API.
  

• I noticed one webhook is created per workspace requesting it. This means that for a monorepo you'll have tons of webhooks (one per workspace), which I believe will not properly scale for my use case (more than 400 workspaces and growing).

  3 workspaces == 3 webhooks and this was just a TK PoC
  

• TFC creates just one webhook for all workspaces (VCS level).

• I believe it could be useful to separate the webhooks endpoint at the ingress-level, so it could be handled in a different way than the API to make it public without any risk.

Regarding the discussion: I don't have access to that Slack workspace. Where could I get an invitation?

Regarding the API docs, thank you for the advice. I was able to check it this time following what you said. However, I was expecting to find there an unauthenticated Swagger endpoint already displaying it, which I believe could be very useful.

[alfespa17]

The invitation link is inside the README @juan-vg

[juan-vg]

Regarding the override branch, it was introduced at #644 and it was already excluded.
To be honest, I don't also fully understand why it was excluded, but I suspect it could be related to

terrakube/api/src/main/java/org/terrakube/api/plugin/state/RemoteTfeService.java

Lines 697 to 701 in fe028bf

	if(workspace.getVcs() != null){
	log.warn("Workspace is using VCS connection, overriding vcs source and branch to run job using a remote plan");
	job.setOverrideBranch("remote-content");
	job.setOverrideSource(sourceTarGz);
	}

(which, again, I honestly don't really understand the logic behind)

[alfespa17]

Regarding the override branch, it was introduced at #644 and it was already excluded. To be honest, I don't also fully understand why it was excluded, but I suspect it could be related to

terrakube/api/src/main/java/org/terrakube/api/plugin/state/RemoteTfeService.java

Lines 697 to 701 in fe028bf

	if(workspace.getVcs() != null){
	log.warn("Workspace is using VCS connection, overriding vcs source and branch to run job using a remote plan");
	job.setOverrideBranch("remote-content");
	job.setOverrideSource(sourceTarGz);
	}

(which, again, I honestly don't really understand the logic behind)

To add it to the API we just need to remove the following line

terrakube/api/src/main/java/org/terrakube/api/rs/job/Job.java

Line 77 in d7c3420

@Exclude

remote-content is just a special branch name that is used internally when we setup the URL for the tar.gz file that is used when running the terraform operation for the CLI driven workflow

[juan-vg]

To add it to the API we just need to remove the following line

terrakube/api/src/main/java/org/terrakube/api/rs/job/Job.java

Line 77 in d7c3420

@Exclude

@alfespa17 PR opened (#1635)

[BenjaminDecreusefond]

Hi @juan-vg !

If you would like a tip we ran into the same issue at our company where one webhook is created for each workspace. Knowing that a single repo has a limitation of 20 webhooks it is not scalable. To work around this issue we made a homemade lambda that triggers and run speculative plans on correct project when their terraform is modified. We manage the different steps like terraform plan running, terraform plan finished or terraform plan failed etc.. through Github check runs that we update depending on where we are in the execution.

But yeah ! Agreed if terrakube could propose that feature natively it would be awesome !