Merge pull request #106 from pndaproject/RELEASE-1.3.0

Release 1.3.0

.gitignore

@@ -9,3 +9,5 @@ pnda_env.yaml
 pnda_env*.sh
 *.tar.gz
 .DS_Store
+cli/logs/*cloud-formation-template*.json
+cli/logs/*cloud-formation-parameters*.json

CHANGELOG.md

@@ -1,6 +1,39 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
+## [1.3.0] 2017-08-01
+### Added
+- PNDA-2375: Isolate PNDA from breaking dependency change
+- PNDA-2676: Support for redhat 7 in the bootstrap scripts. To use redhat set `OS_USER` to `ec2_user` and `imageId` to a redhat 7 AMI in pnda_env.yaml
+- PNDA-2680: adding extra index url in pip configuration
+- PNDA-2691: Add GPG key for nodejs repo
+- PNDA-2706: Refactor template generation
+- PNDA-2708: enable offline installation for python
+- PNDA-2709: Use PNDA_MIRROR for RPMS, DEBS and misc files
+- PNDA-2776: Wait on connectivity to cloud instances before trying to use them
+- PNDA-2842: Add key based access to platform-salt git
+- PNDA-2851: Add script to wait for ec2 grains
+- PNDA-3147: Add dry-run option to CLI that can be used to dry-run the changes to the Cloud Formation stack for create and expand operations.
+- Add online fallback for yum
+
+### Changed
+- PNDA-2446: Place PNDA packages in root of PNDA_MIRROR
+- PNDA-2717: Rename mirror paths
+- PNDA-2802: Refactor saltmaster bootstrap code
+- PNDA-2809: Update m3 instance type defaults to use the latest m4 family instead
+- PNDA-2849: Limit how long we wait for host connectivity
+- Remove minion install from salt-master script
+- Include time and log level in console output
+- Update git version in order to be align with deb mirror script
+- Prioritize local mirror over original repo
+
+### Fixed
+- PNDA-2758: Fix bootstrap for expand
+- PNDA-2803: Remove duplicate salt-master restart commands
+- PNDA-2851: Check for ec2 grains before running salt commands as sometimes the ec2 grain wasn't available when running highstate, but was when checked later on
+- Fix pylint violations
+- Fix issue on easy install configuration
+
 ## [1.2.0] 2017-01-20
 ### Fixed
 - PNDA-2595: xvdc volume mounted on /var/log/pnda instead of /var/log/panda

README.md

@@ -21,17 +21,20 @@ This repository contains resources for launching PNDA on [Amazon Web Services](h
 
 4. Copy ```pnda_env_example.yaml``` to create ```pnda_env.yaml```
 
-5. Edit pnda_env.yaml with an Ubuntu 14.04 image ID (`cloud_formation_parameters.imageId`). This should be the AWS default ubuntu AMI for the region you are provisioning into.
+5. Edit pnda_env.yaml with an Ubuntu 14.04 or Redhat Enterprise Linux 7 image ID (`cloud_formation_parameters.imageId`). This should be the AWS default ubuntu or Redhat AMI for the region you are provisioning into. If using an Ubuntu AMI set `ec2_access.OS_USER` to ubuntu; for a Redhat AMI set `ec2_access.OS_USER` to ec2-user.
 
 6. Edit pnda_env.yaml with [AWS credentials](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html) to use to launch the cloud formation stack (`ec2_access.AWS_ACCESS_KEY_ID` and `ec2_access.AWS_SECRET_ACCESS_KEY`). These credentials should have permissions to launch cloud formation stacks and are only ever stored on the client machine.
 
-7. Edit pnda_env.yaml with a version of platform-salt to use. Platform-salt installs PNDA software on the cloud instances created by the PNDA CLI. A local copy of platform-salt can be used (`platform_salt.PLATFORM_SALT_LOCAL`), or a remote git URI to be cloned from the cloud during provisioning (`platform_salt.PLATFORM_GIT_REPO_URI` and `platform_salt.PLATFORM_GIT_BRANCH`). Ensure that the local clone of platform-salt or the `PLATFORM_GIT_BRANCH` correspond to the same release/x.x.x tag that this pnda-aws-templates repository was cloned at.
+7. Edit pnda_env.yaml with a version of platform-salt to use. Platform-salt installs PNDA software on the cloud instances created by the PNDA CLI. There are two main options:
+   - A local copy of platform-salt can be used by setting (`platform_salt.PLATFORM_SALT_LOCAL`) to the path to the platform-salt folder on the local machine running pnda-cli.py.
+   - A remote git URI to be cloned from the cloud during provisioning (`platform_salt.PLATFORM_GIT_REPO_URI` and `platform_salt.PLATFORM_GIT_BRANCH`).  If authenticated access to `platform_salt.PLATFORM_GIT_REPO_URI` is required then place the ssh key to use, named git.pem, in the top level directory of this repository and also set `platform_salt.PLATFORM_GIT_REPO_HOST` to the hostname of the server.
+   - **Note** Please ensure that the local clone of platform-salt or the `PLATFORM_GIT_BRANCH` correspond to the same release/x.x.x tag that this pnda-aws-templates repository was cloned at.
 
 8. Edit pnda_env.yaml with the names of the buckets to use for PNDA applications (`pnda_application_repo.PNDA_APPS_CONTAINER`) and PNDA archived data (`pnda_data_archive.PNDA_ARCHIVE_CONTAINER`) and the credentials to access them. These credentials should be created for IAM user(s) with access to the specific S3 buckets only as these credentials will be stored in plain text on some of the nodes launched in AWS. For help creating a user with these permissions, please refer [here](s3help.md).
 
-9. Edit pnda_env.yaml with the component package server IP address (`pnda_component_packages.PACKAGES_SERVER_URI`). A component package server provides a webserver for the binaries for the pnda components, the PNDA guide contains instructions on how to set up a component package server.
+9. Edit pnda_env.yaml with the PNDA mirror IP address (`mirrors.PNDA_MIRROR`). The PNDA mirror server provides a webserver for the binaries for the pnda components plus all the third party resources. [This documentation contains instructions](https://github.com/pndaproject/pnda/blob/develop/CREATING_PNDA.md) on how to set up a PNDA mirror.
 
-10. Create [an ssh keypair](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) to use when creating the EC2 instances for PNDA as ```key_name```. Place the private key ```key_name.pem``` in the root of the pnda-aws-templates directory. Ensure that key_name.pem has 0600 permissions.
+10. Create [an ssh keypair](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) to use when creating the EC2 instances for PNDA as ```key_name```. Place the private key ```key_name.pem``` in the root of the pnda-aws-templates directory. Ensure that key_name.pem has 0600 permissions. For example, if the keypair in AWS is 'pnda' then the local private key file should be named 'pnda.pem' and the corresponding cli parameter should be '-s pnda'.
 
 11. Install pip packages required by the CLI 
     ```sh

bootstrap-scripts/base.sh

@@ -1,6 +1,6 @@
 #!/bin/bash -v
 
-# This script runs on all instances except the saltmaster
+# This script runs on all instances
 # It installs a salt minion and mounts the disks
 
 # The pnda_env-<cluster_name>.sh script generated by the CLI should
@@ -9,17 +9,14 @@
 
 set -e
 
-# If system packages are being installed from an offline bundle then download
-# that bundle and make the packages available for installation
-if [ "x$OS_PACKAGE_MIRROR" != "x" ] ; then
-wget ${OS_PACKAGE_MIRROR%/*}/apt-offline.deb
-dpkg -i apt-offline.deb
-wget $OS_PACKAGE_MIRROR
-apt-offline install ${OS_PACKAGE_MIRROR##*/}
-fi
+DISTRO=$(cat /etc/*-release|grep ^ID\=|awk -F\= {'print $2'}|sed s/\"//g)
 
-apt-get update
-apt-get -y install xfsprogs
+if [ "x$DISTRO" == "xubuntu" ]; then
+export DEBIAN_FRONTEND=noninteractive
+apt-get -y install xfsprogs salt-minion=2015.8.11+ds-1
+elif [ "x$DISTRO" == "xrhel" ]; then
+yum -y install xfsprogs wget salt-minion-2015.8.11-1.el7
+fi
 
 # Mount the log volume, this is always xvdc
 if [ -b /dev/xvdc ];
@@ -29,9 +26,9 @@ then
    mkfs.xfs -f /dev/xvdc
    mkdir -p /var/log/pnda
    sed -i "/xvdc/d" /etc/fstab
-   echo "/dev/xvdc /var/log/pnda auto defaults,nobootwait,comment=cloudconfig 0 2" >> /etc/fstab
+   echo "/dev/xvdc /var/log/pnda auto defaults 0 2" >> /etc/fstab
 fi
-# Mount the other log volumes if they exist, up to 3 more may be mounted but this list could be extended if required
+# Mount the other volumes if they exist, up to 3 more may be mounted but this list could be extended if required
 DISKS="xvdd xvde xvdf"
 DISK_IDX=0
 for DISK in $DISKS; do
@@ -43,18 +40,13 @@ for DISK in $DISKS; do
       mkfs.xfs -f /dev/$DISK
       mkdir -p /data$DISK_IDX
       sed -i "/$DISK/d" /etc/fstab
-      echo "/dev/$DISK /data$DISK_IDX auto defaults,nobootwait,comment=cloudconfig 0 2" >> /etc/fstab
+      echo "/dev/$DISK /data$DISK_IDX auto defaults 0 2" >> /etc/fstab
       DISK_IDX=$((DISK_IDX+1))
    fi
 done
 cat /etc/fstab
 mount -a
 
-# Install the salt minion
-export DEBIAN_FRONTEND=noninteractive
-wget -O install_salt.sh https://bootstrap.saltstack.com
-sh install_salt.sh -D -U stable 2015.8.11
-
 # Set the master address the minion will register itself with
 cat > /etc/salt/minion <<EOF
 master: $PNDA_SALTMASTER_IP
@@ -67,3 +59,23 @@ pnda:
 
 pnda_cluster: $PNDA_CLUSTER 
 EOF
+
+PIP_INDEX_URL="$PNDA_MIRROR/mirror_python/simple"
+TRUSTED_HOST=$(echo $PIP_INDEX_URL | awk -F'[/:]' '/http:\/\//{print $4}')
+cat << EOF > /etc/pip.conf
+[global]
+index-url=$PIP_INDEX_URL
+trusted-host=$TRUSTED_HOST
+extra-index-url=https://pypi.python.org/simple/
+EOF
+cat << EOF > /root/.pydistutils.cfg
+[easy_install]
+index_url=$PIP_INDEX_URL
+find_links=https://pypi.python.org/simple/
+EOF
+
+if [ "x$DISTRO" == "xrhel" ]; then
+cat >> /etc/cloud/cloud.cfg <<EOF
+preserve_hostname: true
+EOF
+fi

bootstrap-scripts/package-install.sh

@@ -0,0 +1,42 @@
+#!/bin/bash -v
+
+set -ex
+DISTRO=$(cat /etc/*-release|grep ^ID\=|awk -F\= {'print $2'}|sed s/\"//g)
+if [ "x$DISTRO" == "xubuntu" ]; then
+  export DEBIAN_FRONTEND=noninteractive
+  # give the local mirror the first priority 
+  sed -i "1ideb $PNDA_MIRROR/mirror_deb/ ./" /etc/apt/sources.list
+  wget -O - $PNDA_MIRROR/mirror_deb/pnda.gpg.key | apt-key add -
+  (curl -L 'https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key' | apt-key add - ) && echo 'deb [arch=amd64] https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/ trusty-cm5.9.0 contrib' > /etc/apt/sources.list.d/cloudera-manager.list
+  (curl -L 'http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2015.8.11/SALTSTACK-GPG-KEY.pub' | apt-key add - ) && echo 'deb [arch=amd64] http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2015.8.11/ trusty main' > /etc/apt/sources.list.d/saltstack.list
+  (curl -L 'https://deb.nodesource.com/gpgkey/nodesource.gpg.key' | apt-key add - ) && echo 'deb [arch=amd64] https://deb.nodesource.com/node_6.x trusty main' > /etc/apt/sources.list.d/nodesource.list
+  apt-get update
+
+elif [ "x$DISTRO" == "xrhel" ]; then
+
+if [ "x$YUM_OFFLINE" == "x" ]; then
+RPM_EXTRAS=rhui-REGION-rhel-server-extras
+RPM_OPTIONAL=rhui-REGION-rhel-server-optional
+RPM_EPEL=https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+yum install -y $RPM_EPEL
+yum-config-manager --enable $RPM_EXTRAS $RPM_OPTIONAL
+yum install -y yum-plugin-priorities yum-utils 
+PNDA_REPO=${PNDA_MIRROR/http\:\/\//}
+PNDA_REPO=${PNDA_REPO/\//_mirror_rpm}
+yum-config-manager --add-repo $PNDA_MIRROR/mirror_rpm
+yum-config-manager --setopt="$PNDA_REPO.priority=1" --enable $PNDA_REPO
+else
+mkdir -p /etc/yum.repos.d.backup/
+mv /etc/yum.repos.d/* /etc/yum.repos.d.backup/
+yum-config-manager --add-repo $PNDA_MIRROR/mirror_rpm
+fi
+
+
+rpm --import $PNDA_MIRROR/mirror_rpm/RPM-GPG-KEY-redhat-release
+rpm --import $PNDA_MIRROR/mirror_rpm/RPM-GPG-KEY-mysql
+rpm --import $PNDA_MIRROR/mirror_rpm/RPM-GPG-KEY-cloudera
+rpm --import $PNDA_MIRROR/mirror_rpm/RPM-GPG-KEY-EPEL-7
+rpm --import $PNDA_MIRROR/mirror_rpm/SALTSTACK-GPG-KEY.pub
+rpm --import $PNDA_MIRROR/mirror_rpm/RPM-GPG-KEY-CentOS-7
+
+fi