Remote templates support #59

yorinasub17 · 2020-08-27T13:46:40Z

This adds support in boilerplate to invoke remote templates by providing it with go-getter compatible URLs instead of paths. In the process I renamed all the places in the config and CLI flags that was using template-folder with template-url to match what we are expecting.

This means that we can now invoke templates in git, either as dependencies or directly on the CLI.

NOTE: This is a backwards incompatible change, but now (when we are getting ready for usage-patterns v2) is probably a good time to do this.

…ate-folder to template-url.

examples/dependencies-remote/boilerplate.yml

brikis98 · 2020-08-28T07:49:16Z

getter-helper/getter_helper.go

+// temporary folder and returns the path to that folder. If there is a subdir in the template URL, return the combined
+// path as well.
+func DownloadTemplatesToTemporaryFolder(templateUrl string) (string, string, error) {
+	workingDir, err := ioutil.TempDir("", "boilerplate-cache*")


So we download to the system temp dir? We may want to call that out in the docs. Note that Terragrunt used to use the temp folder too, but we moved away from it and to a local .terragrunt-cache because:

The tmp folder often led to long file paths that worked poorly on Windows.

Having the files locally made debugging easier than hunting around in a temp folder.

We could avoid re-downloading the files on subsequent re-runs.

Not sure how many of these matter for boilerplate...

Will we want a --terragrunt-source equivalent for local testing?

Also, what a * at the end of boilerplate-cache there?

Also

Not sure how many of these matter for boilerplate...

I think 1 and 3 matter (I don't see how having access to the templates locally is useful in 2), but I think there is a better solution for (1). I'll look into this.

3 sort of matters, but honestly I don't envision someone invoking the same template many times in one sitting. The use case for boilerplate is to scaffold a new environment/region/account and possibly component, which seems like it is something that you do once a day at most. Vs terragrunt is something you might run multiple times (validate then plan, and you might plan many resources, and then you will have to run apply, and possibly retry because terraform is buggy).

Will we want a --terragrunt-source equivalent for local testing?

For now, this is not necessary because you can only specify the template via CLI. I think we should implement this when we implement the boilerplate config based CLI, where you can give it a config file instead of CLI args.

Also, what a * at the end of boilerplate-cache there?

The * is how you tell ioutil.TempDir where to inject the unique ID in the folder name.

Also ...

Looks like the last comment is incomplete. Was there another concern here with the tmp folder?

Attempt at solving (1) [windows long path issue]: ea516e0

Will try it out on my windows box.

Bad news: I tried to run this on windows, and I failed much earlier - I couldn't git clone the repo 😞

See #60

Not sure how many of these matter for boilerplate...

I think 1 and 3 matter (I don't see how having access to the templates locally is useful in 2), but I think there is a better solution for (1). I'll look into this.

3 sort of matters, but honestly I don't envision someone invoking the same template many times in one sitting. The use case for boilerplate is to scaffold a new environment/region/account and possibly component, which seems like it is something that you do once a day at most. Vs terragrunt is something you might run multiple times (validate then plan, and you might plan many resources, and then you will have to run apply, and possibly retry because terraform is buggy).

I was thinking more of the person testing the code locally. That is, the author of the templates, not the user.

Will we want a --terragrunt-source equivalent for local testing?

For now, this is not necessary because you can only specify the template via CLI. I think we should implement this when we implement the boilerplate config based CLI, where you can give it a config file instead of CLI args.

What about for nested templates within dependencies? Again, thinking of a template author testing their code locally.

Also, what a * at the end of boilerplate-cache there?

The * is how you tell ioutil.TempDir where to inject the unique ID in the folder name.

Ah, got it!

Also ...

Looks like the last comment is incomplete. Was there another concern here with the tmp folder?

Hahah, whoops. Not sure. Hopefully nothing important 😁

Bad news: I tried to run this on windows, and I failed much earlier - I couldn't git clone the repo 😞

See #60

Doh.

Ok I fixed the windows issue by supporting urlencoded characters in the file templates. This allowed me to clone the repo. I also fixed some windows specific test failures that were easy as well. See 89ca2b5

Of course, tests that depend on bash fail (e.g., the shell tests) but I feel pretty good about the windows tests in general. Once this merges into master, I can start looking into setting up CI for windows environments.

I was thinking more of the person testing the code locally. That is, the author of the templates, not the user.

Fair enough. I'll start working on this functionality, but as a separate PR that merges into this branch given that I expect their will be a fair amount of code to add and probably some nuances to discuss that it deserves its own PR scope.

Filed #61 so we don't forget

getter-helper/getter_helper.go

options/options.go

…filepaths

yorinasub17 · 2020-08-30T03:23:24Z

UPDATE: e77854c

Parallelize tests, and split out shell tests to only run on unix.

brikis98

LGTM!

brikis98 · 2020-08-31T08:57:12Z

integration-tests/examples_test.go

-			continue
-		}
+	// Insulate the following parallel tests in a group so that cleanup routines run after all tests are done.
+	t.Run("group", func(t *testing.T) {


Should there be a t.Parallel() in here? If not, does this run sequentially and then all other boilerplate tests after, in parallel?

This needs to be synchronous so that the defer runs after this grouped test. This is a trick for having cleanup routines with parallel subtests (See "Cleaning up after a group of parallel tests" in the subtests knowledge blog entry).

yorinasub17 · 2020-08-31T15:13:16Z

Thanks for review! Will merge this in.

yorinasub17 added 4 commits August 27, 2020 01:06

Add support for fetching remote templates. Additionally, rename templ…

259f244

…ate-folder to template-url.

Fix dependencies-remote example

2fb4a96

Move download templates test to right place

1974dfd

Fix build

46d5336

yorinasub17 requested review from brikis98, bwhaley and robmorgan as code owners August 27, 2020 13:46

brikis98 reviewed Aug 28, 2020

View reviewed changes

yorinasub17 added 4 commits August 28, 2020 14:16

Simplify get routine by removing goroutine

b7750e8

Attempt to solve windows long path issue using UNC

ea516e0

Support windows by allowing urlencoded file paths and canonicalizing …

89ca2b5

…filepaths

Split out shell tests to only run on unix

e77854c

brikis98 approved these changes Aug 31, 2020

View reviewed changes

yorinasub17 merged commit c6198a5 into master Aug 31, 2020

yorinasub17 deleted the yori-remote-templates branch August 31, 2020 15:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remote templates support #59

Remote templates support #59

yorinasub17 commented Aug 27, 2020

brikis98 Aug 28, 2020

yorinasub17 Aug 28, 2020

yorinasub17 Aug 28, 2020

yorinasub17 Aug 29, 2020

brikis98 Aug 29, 2020

brikis98 Aug 29, 2020

yorinasub17 Aug 30, 2020

yorinasub17 Aug 31, 2020

yorinasub17 commented Aug 30, 2020

brikis98 left a comment

brikis98 Aug 31, 2020 •

edited

Loading

yorinasub17 Aug 31, 2020

yorinasub17 commented Aug 31, 2020

Remote templates support #59

Remote templates support #59

Conversation

yorinasub17 commented Aug 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yorinasub17 commented Aug 30, 2020

brikis98 left a comment

Choose a reason for hiding this comment

brikis98 Aug 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yorinasub17 commented Aug 31, 2020

brikis98 Aug 31, 2020 •

edited

Loading