Instructions and notes for creating the AMI for the evomics.org Harvard Workshop 2016 and information on the previous years' AMIs in previous_workflow.md.
This repository has been forked from Guy Leonard's Evomics 2017 repository (https://github.com/guyleonard/evomics_2017). Minor modifications have been made for the Harvard 2016 Workshop, but the majority of the content, including this README is identical to the original repository. At the time of forking, the set up for the Phylogenomics workshop was incomplete, and this repository reflects that.
Install notes for additional software added for Harvard 2016 can be found in: install_notes.md.
For the past two years I have created the AMIs for the Workshop on Genomics a couple of weeks prior to the workshops beginning - usually in my Christmas holiday and also during the Faculty set up week. This year I want to try and get the building of the AMI to be as automated as possible ahead of the workshop. This will also allow students, faculty, PIs, etc, to build/alter their own copies for future workshops, training or personal use. If you plan to make changes for your own use, I highly suggest forking this repository and making changes to your copy (as you will have to supply your own crypted passwords etc). I am happy to accept pull requests for software bugs, updates, fixes, additions etc.
We will be using the latest Ubuntu Linux as our initial AMI, in this case: ami-cf68e0d8 which is the 'us-east-1' copy of Ubuntu Xenial Xerus 16.04 LTS. But you should be able to use any Debian based version of Linux, ansible will stop on errors but it can also be resumed from that point. I am assuming that you have an AWS account and have already initialised the above AMI for this README.
For this we are going to use a series of shell scripts, Ansible and a few other package managers such as apt-get, pip, conda, and gem. You may also wish to read ami_changes.md. Ansible is a simple piece of automation software you can use to install programs and manage systems. You can see what is happening in each stage by looking in the 'main_*.yaml' file (or playbook) for each section (e.g. base, genomics, phylogenomics), it is a plain text file with a list of instructions to install programs, change files, download data etc. Some of these instructions are broken into smaller components called taks (e.g. the Assembly Tutorial from the Workshop on Genomics is a series of programs and files to be installed) and they can be found in the sub-folder "tasks" in each workshop.
To build the AMI for either workshop, you will need to run at least the setup.sh script on your remote Virtual Machine (this can be on an Amazon VM (AMI) or from another provider), and then both sets of "main_*.yaml" playbooks from either or both workshops.
Log in to your virtual machine and run this code:
wget -O- https://raw.githubusercontent.com/guyleonard/evomics_2017/master/setup.sh | bash
Typically, I would now make an Amazon Machine Image of this system and then use that as your "base AMI" for the next steps (this will save time if you run in to any issues).
If you already have a base AMI that you wish to use you can do this instead:
git clone https://github.com/guyleonard/evomics_2017.git
ansible-playbook /home/ubuntu/evomics_2017/base/main.yaml -b -K -c local -i "localhost,"
If you are logged in to your AMI using a key pair then you do not need to enter a password when you are asked for SUDO access. Just press 'enter' and continue.
Run the genomics software playbook fully:
ANSIBLE_NOCOWS=1 ansible-playbook /home/ubuntu/evomics_2017/genomics/main_software.yaml -b -K -c local -i "localhost,"
or for just two tools, e.g. samtools & bwa:
ANSIBLE_NOCOWS=1 ansible-playbook /home/ubuntu/evomics_2017/genomics/main_software.yaml -b -K -c local -i "localhost," --tags samtools,bwa
or for just one tutorial you could do:
ANSIBLE_NOCOWS=1 ansible-playbook /home/ubuntu/evomics_2017/genomics/main_software.yaml -b -K -c local -i "localhost," --tags assembly
Tags can be mixed and matched as you like, they should install their dependencies where I have remembered ;).
There is only one playbook for the data section, but you can run it with tags as before.
ANSIBLE_NOCOWS=1 ansible-playbook /home/ubuntu/evomics_2017/genomics/main_data.yaml -b -K -c local -i "localhost,"
- Hey Guy, why does it say "ANSIBLE_NOCOWS=1" above?
_______________________________________
/ Do you really want to see cows in all \
| the output messages? Like this? Why? |
\ Moooooooo. /
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||