Private mode launch template failing on initial instances

[cisherlock]

When running terraform apply in private mode for the first time, all resources are created, but the launch template fails resulting in Vault not being installed. System log below.

Terminating the vault instances results in new ones being launched, and the launch template succeeds. Is there a solution to this?

[ 48.410507] cloud-init[2974]: Another app is currently holding the yum lock; waiting for it to exit...
[ 48.417840] cloud-init[2974]: The other application is: yum
[ 48.423171] cloud-init[2974]: Memory : 31 M RSS (323 MB VSZ)
[ 48.428140] cloud-init[2974]: Started: Tue Feb 23 12:33:53 0000 - 00:31 ago
[ 48.434808] cloud-init[2974]: State : Sleeping, pid: 3150
[ 85.462738] cloud-init[2974]: One of the configured repositories failed (Unknown),
[ 85.467694] cloud-init[2974]: and yum doesn't have enough cached data to continue. At this point the only
[ 85.480435] cloud-init[2974]: safe thing yum can do is fail. There are a few ways to work "fix" this:
[ 85.492479] cloud-init[2974]: 1. Contact the upstream for the repository and get them to fix the problem.
[ 85.498847] cloud-init[2974]: 2. Reconfigure the baseurl/etc. for the repository, to point to a working
[ 85.503767] cloud-init[2974]: upstream. This is most often useful if you are using a newer
[ 85.508195] cloud-init[2974]: distribution release than is supported by the repository (and the
[ 85.512624] cloud-init[2974]: packages for the previous distribution release still work).
[ 85.516979] cloud-init[2974]: 3. Run the command with the repository temporarily disabled
[ 85.522600] cloud-init[2974]: yum --disablerepo= ...
[ 85.526017] cloud-init[2974]: 4. Disable the repository permanently, so yum won't use it by default. Yum
[ 85.531440] cloud-init[2974]: will then just ignore the repository until you permanently enable it
[ 85.536171] cloud-init[2974]: again or use --enablerepo for temporary usage:
[ 85.544337] cloud-init[2974]: yum-config-manager --disable
[ 85.552124] cloud-init[2974]: or
[ 85.556101] cloud-init[2974]: subscription-manager repos --disable=
[ 85.564114] cloud-init[2974]: 5. Configure the failing repository to be skipped, if it is unavailable.
[ 85.572471] cloud-init[2974]: Note that yum will try to contact the repo. when it runs most commands,
[ 85.584463] cloud-init[2974]: so will have to try and fail each time (and thus. yum will be be much
[ 85.596301] cloud-init[2974]: slower). If it is a very temporary problem though, this is often a nice
[ 85.603881] cloud-init[2974]: compromise:
[ 85.608370] cloud-init[2974]: yum-config-manager --save --setopt=.skip_if_unavailable=true
[ 85.628201] cloud-init[2974]: Cannot find a valid baseurl for repo: amzn2-core/2/x86_64
[ 85.636112] cloud-init[2974]: Could not retrieve mirrorlist http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list error was
[ 85.656343] cloud-init[2974]: 12: Timeout on http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list: (28, 'Connection timed out after 5000 milliseconds')
[ 85.676365] cloud-init[2974]: Feb Feb 23 12:33:53 cloud-init[2974]: util.py[WARNING]: Package upgrade failed
[ 85.696348] cloud-init[2974]: Feb Feb 23 12:33:53 cloud-init[2974]: cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
[ 85.712352] cloud-init[2974]: Feb 23 12:33:53 cloud-init[2974]: util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_package_update_upgrade_install.pyc'>) failed
[ 86.149613] cloud-init[3170]: Cloud-init v. 19.3-5.amzn2 running 'modules:final' at Tue, 23 Feb 2021 12:33:53+0000. Up 86.09 seconds.
[ 86.601825] cloud-init[3170]: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
[ 121.673744] cloud-init[3170]: One of the configured repositories failed (Unknown),

[jcolemorrison]

Hey @cisherlock, thanks a TON for pointing this out. At this moment I don't have an answer for you, but I'll definitely take a look into this (probably later this week/weekend). I've not yet seen the "(Unknown)" thing before, and I also find it curious that it works when the instances come back up due to the autoscaling group.

My first impression is that it's the result of some networking piece, firewall, or ACL not being ready fast enough. This would lead to it timing out when trying to grab these mirrorlists if the instances are being blocked from reaching outwards to grab its packages. Do you have any other information on hand about the failure?

[cisherlock]

That's all the info I have. My thoughts are the same, something isn't being setup quick enough for the instances to have internet access, with both private_mode and operator_mode set to true.

[jcolemorrison]

Hey just wanted to update that I'm planning to put some work on this coming up soon, just been busy with other things.

[jcolemorrison]

If anyone else encounters this again, let me know. When using this with later version of Terraform and Vault, I've not run into it yet.