From eecb7eb44c915f567c6425ae7214e6c647c6fc04 Mon Sep 17 00:00:00 2001 From: Yevgeniy Brikman Date: Mon, 9 May 2016 14:47:23 +0200 Subject: [PATCH 1/2] v1 of gruntwork-install and bootstrap-gruntwork-installer --- README.md | 94 ++++++++++++++++ bootstrap-gruntwork-installer.sh | 186 +++++++++++++++++++++++++++++++ gruntwork-install | 177 +++++++++++++++++++++++++++++ 3 files changed, 457 insertions(+) create mode 100644 README.md create mode 100755 bootstrap-gruntwork-installer.sh create mode 100755 gruntwork-install diff --git a/README.md b/README.md new file mode 100644 index 0000000..de5edcc --- /dev/null +++ b/README.md @@ -0,0 +1,94 @@ +# Gruntwork Installer + +[Gruntwork Script Modules](https://github.com/gruntwork-io/script-modules) is a private repo that contains scripts and +applications developed by [Gruntwork](http://www.gruntwork.io) for common infrastructure tasks such as setting up +continuous integration, monitoring, log aggregation, and SSH access. This repo contains provides a script called +`gruntwork-install` that makes it as easy to install the Gruntwork Script Modules as using apt-get, brew, or yum. + +For example, in your Packer and Docker templates, you can use `gruntwork-install` as follows: + +```bash +gruntwork-install --module-name 'vault-ssh-helper' --tag '0.0.3' +``` + +## Installing gruntwork-install + +```bash +curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash -s --version=0.0.1 +``` + +Notice the `--version` parameter at the end where you specify which version of `gruntwork-install` to install. See the +[releases](/releases) page for all available versions. + +## Using gruntwork-install + +#### Authentication + +Since the [Script Modules](https://github.com/gruntwork-io/script-modules) repo is private, you must set your +[GitHub access token](https://help.github.com/articles/creating-an-access-token-for-command-line-use/) as the +environment variable `GITHUB_OAUTH_TOKEN` so `gruntwork-install` can use it to access the repo: + +```bash +export GITHUB_OAUTH_TOKEN="..." +``` + +#### Options + +Once that environment variable is set, you can run `gruntwork-install` with the following options: + +* `--module-name`: Required. The name of the Script Module to install. Can be any folder within the `modules` directory + of the [Script Modules Repo](https://github.com/gruntwork-io/script-modules). +* `--tag`: Required. The version of the Script Module to install. Follows the syntax described at [Fetch Version + Constraint Operators](https://github.com/gruntwork-io/fetch#version-constraint-operators). +* `--branch`: Optional. Download the latest commit from this branch. This is an alternative to `--tag` for development + purposes. +* `--module-param`: Optional. A key-value pair of the format `key=value` you wish to pass to the module as a parameter. + May be used multiple times. See the documentation for each module to find out what parameters it accepts. +* `--help`: Optional. Show the help text and exit. + +#### Examples + +Here is how you could use `gruntwork-install` to install the `vault-ssh-helper` module, version `0.0.3`: + +```bash +gruntwork-install --module-name 'vault-ssh-helper' --tag '0.0.3' +``` + +And here is an example of using `--module-param` to pass two custom parameters to the `vault-ssh-helper` module: + +```bash +gruntwork-install --module-name 'vault-ssh-helper' --tag '0.0.3' --module-param 'install-dir=/opt/vault-ssh-helper' --module-param 'owner=ubuntu' +``` + +And finally, to put all the pieces together, here is an example of a Packer template that installs `gruntwork-install` +and then uses it to install several modules: + +```json +{ + "variables": { + "github_auth_token": "{{env `GITHUB_OAUTH_TOKEN`}}" + }, + "builders": [{ + "ami_name": "gruntwork-install-example-{{isotime | clean_ami_name}}", + "instance_type": "t2.micro", + "region": "us-east-1", + "type": "amazon-ebs", + "source_ami": "ami-fce3c696", + "ssh_username": "ubuntu" + }], + "provisioners": [{ + "type": "shell", + "inline": "curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash -s --version=0.0.1" + },{ + "type": "shell", + "inline": [ + "gruntwork-install --module-name 'vault-ssh-helper' --tag '~>0.0.4' --module-param 'install-dir=/opt/vault-ssh-helper' --module-param 'owner=ubuntu'", + "gruntwork-install --module-name 'cloudwatch-log-aggregation' --tag '~>0.0.4'", + "gruntwork-install --module-name 'build-helpers' --tag '~>0.0.4'" + ], + "environment_vars": [ + "GITHUB_OAUTH_TOKEN={{user `github_auth_token`}}" + ] + }] +} +``` \ No newline at end of file diff --git a/bootstrap-gruntwork-installer.sh b/bootstrap-gruntwork-installer.sh new file mode 100755 index 0000000..2c3b97b --- /dev/null +++ b/bootstrap-gruntwork-installer.sh @@ -0,0 +1,186 @@ +#!/bin/bash +# +# A bootstrap script to install the Gruntwork Installer. +# +# Why: +# +# The goal of the Gruntwork Installer is to make make installing Gruntwork Script Modules feel as easy as installing a +# package using apt-get, brew, or yum. However, something has to install the Gruntwork Installer first. One option is +# for each Gruntwork client to do so manually, which would basically entail copying and pasting all the code below. +# This is tedious and would give us no good way to push updates to this bootstrap script. +# +# So instead, we recommend that clients use this tiny bootstrap script as a one-liner: +# +# curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash -s --version=0.0.12 +# +# You can copy this one-liner into your Packer and Docker templates and immediately after, start using the +# gruntwork-install command. + +set -e + +readonly DEFAULT_FETCH_VERSION="v0.0.3" +readonly FETCH_DOWNLOAD_URL_BASE="https://github.com/gruntwork-io/fetch/releases/download" +readonly FETCH_INSTALL_PATH="/usr/local/bin/fetch" + +readonly GRUNTWORK_INSTALLER_DOWNLOAD_URL_BASE="https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer" +readonly GRUNTWORK_INSTALLER_INSTALL_PATH="/usr/local/bin/gruntwork-install" +readonly GRUNTWORK_INSTALLER_SCRIPT_NAME="gruntwork-install" + +function print_usage { + echo + echo "Usage: bootstrap-gruntwork-installer.sh [OPTIONS]" + echo + echo "A bootstrap script to install the Gruntwork Installer ($GRUNTWORK_INSTALLER_SCRIPT_NAME)." + echo + echo "Options:" + echo + echo -e " --version\t\tRequired. The version of $GRUNTWORK_INSTALLER_SCRIPT_NAME to install (e.g. 0.0.3)." + echo -e " --fetch-version\tOptional. The version of fetch to install. Default: $DEFAULT_FETCH_VERSION." + echo + echo "Examples:" + echo + echo " Install version 0.0.3:" + echo " bootstrap-gruntwork-installer.sh --version=0.0.3" + echo + echo " One-liner to download this bootstrap script from GitHub and run it to install version 0.0.3:" + echo " curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash -s --version=0.0.3" +} + +function command_exists { + local readonly cmd="$1" + type "$cmd" > /dev/null 2>&1 +} + +function download_url_to_file { + local readonly url="$1" + local readonly file="$2" + + echo "Downloading $url to $file" + if $(command_exists "curl"); then + local readonly status_code=$(curl -L -s -w '%{http_code}' -o "$file" "$url") + if [[ "$status_code" != "200" ]]; then + echo "ERROR: Expected status code 200 but got $status_code when downloading $url" + exit 1 + fi + else + echo "ERROR: curl is not installed. Cannot download $url." + exit 1 + fi +} + +function string_contains { + local readonly str="$1" + local readonly contains="$2" + + [[ "$str" == *"$contains"* ]] +} +# http://stackoverflow.com/a/2264537/483528 +function to_lower_case { + tr '[:upper:]' '[:lower:]' +} + +function get_os_name { + uname | to_lower_case +} + +function get_os_arch { + uname -m +} + +function get_os_arch_gox_format { + local readonly arch=$(get_os_arch) + + if $(string_contains "$arch" "64"); then + echo "amd64" + elif $(string_contains "$arch" "386"); then + echo "386" + elif $(string_contains "$arch" "arm"); then + echo "arm" + fi +} + +function download_and_install { + local readonly url="$1" + local readonly install_path="$2" + + download_url_to_file "$url" "$install_path" + chmod 0755 "$install_path" +} + +function install_fetch { + local readonly install_path="$1" + local readonly version="$2" + + local readonly os=$(get_os_name) + local readonly os_arch=$(get_os_arch_gox_format) + + if [[ -z "$os_arch" ]]; then + echo "ERROR: Unrecognized OS architecture: $(get_os_arch)" + exit 1 + fi + + echo "Installing fetch version $version to $install_path" + local readonly url="${FETCH_DOWNLOAD_URL_BASE}/${version}/fetch_${os}_${os_arch}" + download_and_install "$url" "$install_path" +} + +function install_gruntwork_installer { + local readonly install_path="$1" + local readonly version="$2" + + echo "Installing $GRUNTWORK_INSTALLER_SCRIPT_NAME version $version to $install_path" + local readonly url="${GRUNTWORK_INSTALLER_DOWNLOAD_URL_BASE}/${version}/${GRUNTWORK_INSTALLER_SCRIPT_NAME}" + download_and_install "$url" "$install_path" +} + +function assert_not_empty { + local readonly arg_name="$1" + local readonly arg_value="$2" + + if [[ -z "$arg_value" ]]; then + echo "ERROR: The value for '$arg_name' cannot be empty" + print_usage + exit 1 + fi +} + +function bootstrap { + local fetch_version="$DEFAULT_FETCH_VERSION" + local installer_version="" + + while [[ $# > 0 ]]; do + local key="$1" + + case "$key" in + --version) + installer_version="$2" + shift + ;; + --fetch-version) + fetch_version="$2" + shift + ;; + --help) + print_usage + exit + ;; + *) + echo "ERROR: Unrecognized option: $key" + print_usage + exit 1 + ;; + esac + + shift + done + + assert_not_empty "--version" "$installer_version" + assert_not_empty "--fetch-version" "$fetch_version" + + echo "Installing $GRUNTWORK_INSTALLER_SCRIPT_NAME..." + install_fetch "$FETCH_INSTALL_PATH" "$fetch_version" + install_gruntwork_installer "$GRUNTWORK_INSTALLER_INSTALL_PATH" "$installer_version" + echo "Success!" +} + +bootstrap "$@" \ No newline at end of file diff --git a/gruntwork-install b/gruntwork-install new file mode 100755 index 0000000..e0a4301 --- /dev/null +++ b/gruntwork-install @@ -0,0 +1,177 @@ +#!/bin/bash +# +# This script downloads a Gruntwork Script Module from https://github.com/gruntwork-io/script-modules using fetch +# (https://github.com/gruntwork-io/fetch), and then runs it. The main motivation in writing it is to make installing +# Gruntwork Script Modules feel as easy as installing a package using apt-get, brew, or yum. +# +# Note that if the user specifies neither --tag nor --branch, the latest tag is downloaded. +# + +set -e + +readonly SCRIPT_MODULES_REPO="https://github.com/gruntwork-io/script-modules" +readonly MODULES_DIR="modules" + +readonly MODULES_DOWNLOAD_DIR="/tmp/gruntwork-script-modules" +readonly MODULE_INSTALL_FILE_NAME="install.sh" + +function print_usage { + echo + echo "Usage: gruntwork-install [OPTIONS]" + echo + echo "Download a Gruntwork Script Module from https://github.com/gruntwork-io/script-modules and install it." + echo + echo "Options:" + echo + echo -e " --module-name\t\tRequired. The name of the Script Module to install. Must be a folder within the $MODULES_DIR directory of $SCRIPT_MODULES_REPO." + echo -e " --tag\t\t\tRequired. The version of the Script Module to install. Follows the syntax described at https://github.com/gruntwork-io/fetch#version-constraint-operators." + echo -e " --branch\t\tOptional. Download the latest commit from this branch. This is an alternative to --tag for development purposes." + echo -e " --module-param\tOptional. A key-value pair of the format key=value you wish to pass to the module as a parameter. May be used multiple times." + echo + echo "Example:" + echo + echo " gruntwork-install --module-name 'vault-ssh-helper' --tag '~>0.0.3' --module-param 'install-dir=/opt/vault-ssh-helper' --module-param 'owner=ubuntu'" + echo +} + +# Assert that a given binary is installed on this box +function assert_is_installed { + local readonly name="$1" + + if [[ ! $(command -v ${name}) ]]; then + echo "ERROR: The binary '$name' is required by this script but is not installed or in the system's PATH." + exit 1 + fi +} + +function assert_not_empty { + local readonly arg_name="$1" + local readonly arg_value="$2" + + if [[ -z "$arg_value" ]]; then + echo "ERROR: The value for '$arg_name' cannot be empty" + print_usage + exit 1 + fi +} + +function assert_env_var_not_empty { + local readonly var_name="$1" + local readonly var_value="${!var_name}" + + if [[ -z "$var_value" ]]; then + echo "ERROR. Required environment $var_name not set." + exit 1 + fi +} + +# Download the files of the given Script Module using fetch (https://github.com/gruntwork-io/fetch) +function fetch_script_module { + local readonly module_name="$1" + local readonly tag="$2" + local readonly branch="$3" + local readonly github_token="$4" + local readonly download_path="$5" + + # We want to make sure that all folders down to $download_path/$module_name exists, but that $download_path/$module_name itself is empty. + mkdir -p "$download_path/$module_name/" + rm -Rf "$download_path/$module_name/" + + # Note that fetch can safely handle blank arguments for --tag or --branch + # If both --tag and --branch are specified, --branch will be used + # TODO: fetch should read GITHUB_OAUTH_TOKEN as an environment variable too + fetch --repo="$SCRIPT_MODULES_REPO" --tag="$tag" --branch="$branch" --github-oauth-token="$github_token" "/modules/$module_name" "$download_path/$module_name" >/dev/null +} + +# Validate that at least one file was downloaded from the module; otherwise throw an error. +function validate_module { + local readonly module_name="$1" + local readonly download_path="$2" + + if [[ ! -e "$download_path/$module_name" ]]; then + echo "ERROR: No files were downloaded. Are you sure \"$module_name\" is a valid Script Module in $SCRIPT_MODULES_REPO?" + exit 1 + fi +} + +# Take in an array of key-val pairs in this format: +# module_params[0] = "key1=value1" +# module_params[1] = "key2=value2" +# .. and convert it to the format that Gruntwork bash scripts expect: +# --key1 value1 --key2 value2 +function convert_module_params_format { + local readonly module_params=("${@}") + + module_params_formatted="" + for key_val_expression in "${module_params[@]}"; do + key="${key_val_expression%=*}" + val="${key_val_expression#*=}" + module_params_formatted="--${key} ${val} ${module_params_formatted}" + done + + echo $module_params_formatted +} + +function run_module { + local readonly module_name="$1" + shift + local readonly module_params=($@) + local readonly module_params_formatted=$(convert_module_params_format "${module_params[@]}") + + chmod -R u+x "${MODULES_DOWNLOAD_DIR}/${module_name}" + eval "${MODULES_DOWNLOAD_DIR}/${module_name}/${MODULE_INSTALL_FILE_NAME} ${module_params_formatted}" +} + +function install_script_module { + local tag="" + local branch="" + local module_name="" + local module_params=() + + while [[ $# > 0 ]]; do + local key="$1" + + case "$key" in + --tag) + tag="$2" + shift + ;; + --branch) + branch="$2" + shift + ;; + --module-name) + module_name="$2" + shift + ;; + --module-param) + module_param="$2" + module_params+=("$module_param") + shift + ;; + --help) + print_usage + exit + ;; + *) + echo "ERROR: Unrecognized option: $key" + print_usage + exit 1 + ;; + esac + + shift + done + + assert_is_installed fetch + assert_not_empty "--module-name" "$module_name" + assert_env_var_not_empty "GITHUB_OAUTH_TOKEN" + + echo "Installing $module_name..." + fetch_script_module "$module_name" "$tag" "$branch" "$GITHUB_OAUTH_TOKEN" "$MODULES_DOWNLOAD_DIR" + validate_module "$module_name" "$MODULES_DOWNLOAD_DIR" + run_module "$module_name" "${module_params[@]}" + echo "Success!" +} + +install_script_module "$@" \ No newline at end of file From 26768d35f0aeecd6ed385f4ec1960b67e99ca1a6 Mon Sep 17 00:00:00 2001 From: Yevgeniy Brikman Date: Mon, 9 May 2016 15:14:07 +0200 Subject: [PATCH 2/2] Add discussion of piping scripts into bash --- README.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index de5edcc..986bf86 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,8 @@ curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/mast Notice the `--version` parameter at the end where you specify which version of `gruntwork-install` to install. See the [releases](/releases) page for all available versions. +For paranoid security folks, see [is it safe to pipe URLs into bash?](#is-it-safe-to-pipe-urls-into-bash) below. + ## Using gruntwork-install #### Authentication @@ -91,4 +93,33 @@ and then uses it to install several modules: ] }] } -``` \ No newline at end of file +``` + +## Is it safe to pipe URLs into bash? + +Are you worried that our install instructions tell you to pipe a URL into bash? Although this approach has seen some +[backlash](https://news.ycombinator.com/item?id=6650987), we believe that the convenience of a one-line install +outweighs the minimal security risks. Below is a brief discussion of the most commonly discussed risks and what you can +do about them. + +#### Risk #1: You don't know what the script is doing, so you shouldn't blindly execute it. + +This is true of *all* installers. For example, have you ever inspected the install code before running `apt-get install` +or `brew install` or double cliking a `.dmg` or `.exe` file? If anything, a shell script is the most transparent +installer out there, as it's one of the few that allows you to inspect the code (feel free to do so, as this script is +open source!). The reality is that you either trust the developer or you don't. And eventually, you automate the +install process anyway, at which point manual inspection isn't a possibility anyway. + +#### Risk #2: The download URL could be hijacked for malicious code. + +This is unlikely, as it is an https URL, and your download program (e.g. `curl`) should be verifying SSL certs. That +said, Certificate Authorities have been hacked in the past, and if that is a major concern for you, feel free to copy +the bootstrap code into your own codebase and execute it from there. + +#### Risk #3: The script may not download fully and executing it could cause catastrophic errors. + +We wrote our bootstrap script as a series of bash functions that are only executed by the very last line of the script. +Therefore, if the script doesn't fully download, the worst that'll happen when you execute it is a harmless syntax +error. + +