diff --git a/lectures/week_01/bfx_workshop_01_overview.ipynb b/lectures/week_01/bfx_workshop_01_overview.ipynb index 1127c0c..53524f6 100644 --- a/lectures/week_01/bfx_workshop_01_overview.ipynb +++ b/lectures/week_01/bfx_workshop_01_overview.ipynb @@ -6,7 +6,7 @@ "source": [ "# BFX Workshop Prerequisite Review\n", "\n", - "Welcome back to the Fall 2021 and Spring 2022 Bioinformatics (BFX) Workshop. \n" + "Welcome back to the Fall 2022 and Spring 2023 Bioinformatics (BFX) Workshop. \n" ] }, { @@ -17,13 +17,16 @@ "\n", "Whether on a macOS or Linux operating system, the Unix family of operating systems all have a similar look and feel. There are standard commands for navigating a filesystem and utilities for performing common tasks on files and directories. This workshop does not specifically cover Unix commands, but a few good resources:\n", "\n", - "* [This](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Finfo&reset=1&id=402) virtual workshop is offered in collaboration with WUIT's Research Infrastructure Services (RIS) and will introduce the Unix command line interface, the Unix file system, and basic Unix commands. \n", + "* [Computing Basics – The Unix Shell and Command Line Basics](https://becker.wustl.edu/civicrm/?page=CiviCRM&q=civicrm/event/info&reset=1&id=533) is a virtual workshop offered by [Bernard Becker Medical Library](https://becker.wustl.edu/) in collaboration with WUIT's Research Infrastructure Services (RIS) and will introduce the Unix command line interface, the Unix file system, and basic Unix commands.\n", " * Attendees will learn to create, explore and manage Unix folders and files.\n", - "* [This](https://www.linkedin.com/learning/unix-for-mac-os-x-users) video tutorial should be easy to follow along on a macOS.\n", - " * Although several years old and designed for Max OS X, not much has changed with Unix.\n", - " * LinkedIn Learning is Free to all WUSTL faculty, students and staff. \n", - "* For traditional users, the classic look and feel of [this](http://www.ee.surrey.ac.uk/Teaching/Unix/unixintro.html) tutorial will provide an introduction to UNIX commands, utilities, and navigation tools.\n", - "* Numerous other Linux/UNIX tutorials exist, feel free to post your favorite in the #bfx_workshop slack channel.\n" + " * [Register Now](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Fregister&id=533&reset=1) September 15th, 2022 10:00 AM - 11:30 AM [Add to Calendar](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Fical&reset=1&id=533)\n", + " * Do you have questions about this event? Contact ndonwimaze at wustl.edu\n", + "* [LinkedIn Learning: Unix for Mac OS X Users](https://www.linkedin.com/learning/unix-for-mac-os-x-users) is an easy to follow macOS video tutorial introducing users to the Unix features shared with other Linux/Unix based Operation Systems (OS).\n", + " * Although the video is several years old and designed for Max OS X specifically, not much has changed with Unix.\n", + " * All current Washington University students, faculty, and staff have access to LinkedIn Learning’s resources, which cover a broad range of topics.\n", + " * Please visit this [WUSTL IT page for LinkedIn Learning login information](https://it.wustl.edu/items/linkedin-learning/).\n", + "* For a more traditional user experience, the classic look and feel of [this UNIX Introduction](http://www.ee.surrey.ac.uk/Teaching/Unix/unixintro.html) will provide a background on the Operation System itself as well as introductions to UNIX commands, utilities, and navigation tools.\n", + "* Numerous other Linux/UNIX tutorials exist, feel free to post your favorite in the [#bfx-workshop](https://ictsprecisionhealth.slack.com/archives/C040Q704WS2) Slack channel.\n" ] }, { @@ -32,9 +35,7 @@ "source": [ "## Software you'll need to have installed:\n", "\n", - "NOTE: There are numerous ways to install Python, R, Notebooks, etc. In fact, you will see various ways of installing software and configuring your environment throughout the workshop. Each presenter and/or software toolkit may require or suggest a different approach. The following installs are a minimum list of steps to both load, execute, and edit the Jupyter Notebooks used for sharing material and instructions used throughout this workshop.\n", - "\n", - "[This](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Finfo&reset=1&id=403) virtual workshop is offered in collaboration with WUIT's Research Infrastructure Services (RIS). The workshop will introduce text editors, shell and batch scripts, Docker container technology, and the ssh protocol. Attendees will learn how to write, edit and run shell scripts, how to create a simple Docker container, and how to connect to the RIS compute cluster." + "There are numerous ways to install Python, R, Notebooks, etc. In fact, you will see various ways of installing software and configuring your environment throughout the workshop. Each presenter and/or software toolkit may require or suggest a different approach. The Install sections below, ex. Miniconda, Jupyter, Git, R, JAVA, Docker, etc. are one list of requirements used to load, execute, and edit Jupyter Notebooks for completing some workshop tutorials on local compute devices, ie. Mac OS X laptop or other Unix/Linux OS personal device. For Windows OS, tutorials may work as-is, but are likely untested on Windows, Unix/Linux Subsystems, Dual Boot OS, or Unix/Linux OS simulation software. " ] }, { @@ -44,10 +45,11 @@ "### Miniconda\n", "Conda is a package manager that works on Windows, Linux, and macOS. We will use a lightweight version of the package manager called Miniconda to install packages and manage the system environment. This tool should help minimize dependencies and differences in system-level libraries or operatings system across the participants in the BFX Workshop.\n", "\n", - "Available installers for Miniconda can be found [here](https://docs.conda.io/en/latest/miniconda.html#miniconda) through the official Conda project documentation. \n", + "Available installers for Miniconda can be found [here](https://docs.conda.io/en/latest/miniconda.html#miniconda) through the official Conda project documentation. NOTE: Please choose the latest version of Python, ex. 3.9, although some users may already be using Python 3.8 or 3.7. Ignore Python 2 installers and please update existing Python 2 installs. \n", "\n", - "For macOS, assumed to be the most common OS used in this course, the suggested installer is circled in red on the image of the avaiable installers:\n", - "![macOS Miniconda Installers](https://raw.githubusercontent.com/genome/bfx_workshop/master/archive/images/miniconda_macos_installers.png)\n", + "For macOS, assumed to be the most common OS used in this course, a Python 3.8 installer (from v2020-2021) is circled in red on the image of the avaiable installers (Please use 3.9 or the latest when installing for the first time):\n", + "![macOS Miniconda Installers](https://github.com/genome/bfx-workshop/raw/master/archive/v2020-2021/images/miniconda_macos_installers.png)\n", + "UPDATE: You will need to choose an Installer appropriate for your Mac chipset, ex. Intel x86 vs. Apple M1 architecture\n", "\n", "Once you've complete the install process, open a fresh terminal and check that conda is now installed by listing the Help menu:\n", "\n", @@ -56,7 +58,7 @@ "```\n", "\n", "An example image of what a terminal might look like when running the above command:\n", - "![Example Conda Help](https://raw.githubusercontent.com/genome/bfx_workshop/master/archive/images/conda_help.png)" + "![Example Conda Help](https://github.com/genome/bfx-workshop/raw/master/archive/v2020-2021/images/conda_help.png)" ] }, { @@ -75,8 +77,8 @@ "python -V\n", "```\n", "\n", - "As of September 7th, 2021, the base conda python version is 3.8.11, see the example image below:\n", - "![Example Python Version](https://raw.githubusercontent.com/genome/bfx_workshop/master/archive/images/python_version.png)\n" + "As of September 8th, 2022, the base conda Python 3.8 version is 3.8.13, see the example image below from 2020 (v3.8.3):\n", + "![Example Python Version](https://github.com/genome/bfx-workshop/raw/master/archive/v2020-2021/images/python_version.png)\n" ] }, { @@ -138,11 +140,11 @@ "#### Git Exercise\n", "Let's clone the class git repository from GitHub, see [link](https://github.com/genome/bfx_workshop).\n", "\n", - "![Example GitHub Clone/Download Code](https://raw.githubusercontent.com/genome/bfx_workshop/master/archive/images/github_clone_repo.png)\n", + "![Example GitHub Clone/Download Code](https://github.com/genome/bfx-workshop/raw/master/archive/v2020-2021/images/github_clone_repo.png)\n", "\n", "NOTE: You have the option of using either HTTPS or SSH to clone the class git repo. Below is an example of using SSH.\n", "\n", - "![Example Git Clone Commmand](https://raw.githubusercontent.com/genome/bfx_workshop/master/archive/images/github_clone_cmd.png)\n" + "![Example Git Clone Commmand](https://github.com/genome/bfx-workshop/raw/master/archive/v2020-2021/images/github_clone_cmd.png)\n" ] }, { @@ -196,53 +198,18 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { "scrolled": true }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[34m94334a6a-b3c4-4a2b-ae31-5c4bdaef2021\u001b[m\u001b[m/\r\n", - "\u001b[34mApplications\u001b[m\u001b[m/\r\n", - "\u001b[34mBox Sync\u001b[m\u001b[m/\r\n", - "\u001b[34mCreative Cloud Files\u001b[m\u001b[m/\r\n", - "\u001b[34mDesktop\u001b[m\u001b[m/\r\n", - "\u001b[34mDocuments\u001b[m\u001b[m/\r\n", - "\u001b[34mDownloads\u001b[m\u001b[m/\r\n", - "\u001b[34mLibrary\u001b[m\u001b[m/\r\n", - "MFN2_Part1.png\r\n", - "MFN2_Part2.png\r\n", - "MFN2_Part3.png\r\n", - "\u001b[34mMovies\u001b[m\u001b[m/\r\n", - "\u001b[34mMusic\u001b[m\u001b[m/\r\n", - "NCBI_SARS-CoV-2.fa\r\n", - "NCBI_SARS-CoV-2.fa.fai\r\n", - "\u001b[34mOneDrive - Washington University in St. Louis\u001b[m\u001b[m/\r\n", - "\u001b[34mPictures\u001b[m\u001b[m/\r\n", - "\u001b[34mPublic\u001b[m\u001b[m/\r\n", - "\u001b[34mbashrc.d\u001b[m\u001b[m/\r\n", - "\u001b[34mbd0e9f28-8708-4288-9beb-79b59a91285b\u001b[m\u001b[m/\r\n", - "\u001b[34md293fe48-db49-41f0-8d19-1a5caa12721d\u001b[m\u001b[m/\r\n", - "\u001b[34mf2e01a48-b2a0-4af0-8aba-8ecdc8c1293e\u001b[m\u001b[m/\r\n", - "\u001b[34mgit\u001b[m\u001b[m/\r\n", - "\u001b[34mgoogle-cloud-sdk\u001b[m\u001b[m/\r\n", - "\u001b[34migv\u001b[m\u001b[m/\r\n", - "\u001b[34mminiconda\u001b[m\u001b[m/\r\n", - "miniconda.sh\r\n", - "\u001b[34mopt\u001b[m\u001b[m/\r\n" - ] - } - ], + "outputs": [], "source": [ "ls" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": {}, "outputs": [], "source": [ @@ -251,7 +218,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "metadata": {}, "outputs": [ { @@ -268,15 +235,15 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "metadata": {}, "outputs": [ { "ename": "SyntaxError", - "evalue": "invalid syntax (, line 1)", + "evalue": "invalid syntax (568906501.py, line 1)", "output_type": "error", "traceback": [ - "\u001b[0;36m File \u001b[0;32m\"\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m echo \"Hello World\"\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n" + "\u001b[0;36m Input \u001b[0;32mIn [6]\u001b[0;36m\u001b[0m\n\u001b[0;31m echo \"Hello World\"\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n" ] } ], @@ -286,17 +253,9 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 8, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Hello World\r\n" - ] - } - ], + "outputs": [], "source": [ "!echo \"Hello World\"" ] @@ -307,20 +266,36 @@ "source": [ "## Compute and Storage Access\n", "\n", - "The remainder of this tutorial includes resources to ensure all participants have access to a High Performance Compute (HPC) cluster.\n", + "### Google Cloud\n", + "***UNDER CONSTRUCTION***\n", + "\n", + "* [WUIT Google Cloud](https://it.wustl.edu/services/cloud-computing/google-cloud-platform/)\n", + "* [Google Cloud Console](https://console.cloud.google.com/)\n", + "\n", + "\n", + "### WashU Local Compute and Storage\n", + "For additional information on Computing Basics with WUIT's Research Infrastructure Services (RIS) supported [Scientific Compute Platform](https://ris.wustl.edu/services/compute/), please see the following workshops offered through [Bernard Becker Medical Library](https://becker.wustl.edu/):\n", "\n", - "HPC Prerequisites:\n", + "* [Computing Basics – Software Applications Needed to Work in High Performance Computing Environments](https://becker.wustl.edu/civicrm/?page=CiviCRM&q=civicrm/event/info&reset=1&id=535) is a virtual workshop offered by [Bernard Becker Medical Library](https://becker.wustl.edu/) in collaboration with WUIT's Research Infrastructure Services (RIS) and will introduce text editors, shell and batch scripts, Docker container technology, and the ssh protocol. \n", + " * Attendees will learn how to write, edit and run shell scripts, how to create a simple Docker container, and how to connect to the RIS compute platform.\n", + " * [Register Now](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Fregister&id=535&reset=1) September 22nd, 2022 10:00 AM - 11:30 AM [Add to Calendar](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Fical&reset=1&id=535)\n", + " * Do you have questions about this event? Contact ndonwimaze at wustl.edu\n", + "* [Computing Basics – Submitting Jobs to the RIS Scientific Compute Platform](https://becker.wustl.edu/civicrm/?page=CiviCRM&q=civicrm/event/info&reset=1&id=534) is a virtual workshop is offered by [Bernard Becker Medical Library](https://becker.wustl.edu/) in collaboration with WUIT's Research Infrastructure Services (RIS) and will introduce basic commands for submitting jobs to the queue on RIS compute platform. \n", + " * Attendees will learn how to submit and run jobs/tasks using Docker and queuing system commands.\n", + " * NOTE: This event is currently full.\n", + " * September 29th, 2022 10:00 AM - 11:30 AM [Add to Calendar](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Fical&reset=1&id=534)\n", + " * Do you have questions about this event? Contact ndonwimaze at wustl.edu\n", "\n", "* The home for RIS docs supporting the compute1/storage1 environment: https://docs.ris.wustl.edu/\n", " * A complete list of RIS workshop recordings: https://docs.ris.wustl.edu/doc/compute/compute-workshops.html\n", - "* [This](https://becker.wustl.edu/civicrm/?civiwp=CiviCRM&q=civicrm%2Fevent%2Finfo&reset=1&id=404) virtual workshop is offered in collaboration with WUIT's Research Infrastructure Services (RIS) and will introduce basic commands for submitting jobs on RIS compute cluster. Attendees will learn how to submit jobs to RIS compute cluster’s queuing system as well as interactively, using Docker.\n" + "* Virtual workshops are offered in collaboration with WUIT's Research Infrastructure Services (RIS) and will introduce basic commands for submitting jobs on RIS compute cluster. Please see ##\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Virtual Private Network (VPN)\n", + "#### Virtual Private Network (VPN)\n", "This course will at times require access to the compute1/storage1 High Perfomance Computing & Storage solutions managed by the Research Infrastructure Services (RIS) team through WUSTL IT. In order to log into this compute environment, first one needs access to the WUSM VPN. For more information, please see the VPN section of this [link](https://it.wustl.edu/items/connect/)." ] }, @@ -328,14 +303,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Scientific Compute/Storage Platform\n", + "#### Scientific Compute/Storage Platform\n", "\n", "This workshop will cover workflows and pipelines that use the compute1/storage1 environment. This compute and storage environment is not required for all workshops. However, there is a complete RIS seminar series from April 2020 available [here](https://www.youtube.com/playlist?list=PLc5dxOEco26RhSbhBaRLeZoUFOTn7GM-Y) if you need more instruction than listed below.\n", "\n", - "#### Getting Started\n", + "##### Getting Started\n", "Please see the \"Getting Started\" section of this [link](https://ris.wustl.edu/services/compute/) if you do not have access to the compute1 platform.\n", "\n", - "#### Getting Connected\n", + "##### Getting Connected\n", "If you have access, but are unsure how to connect to compute1, please see this [link](https://confluence.ris.wustl.edu/display/ITKB/Compute+Quick+Start#ComputeQuickStart-1.GettingConnected). \n", "\n", "NOTE: The above link is only accessible from within WUSM's network.\n" @@ -356,17 +331,16 @@ "source": [ "## Homework Assignments\n", "\n", - "Please attempt the following exercises on your own and post questions to the mgibio #bfx_workshop Slack channel.\n", + "Please attempt the following exercises on your own and post questions to the ICTS Precision Health [#bfx-workshop](https://ictsprecisionhealth.slack.com/archives/C040Q704WS2) Slack channel.\n", "\n", - "1. Ensure you can login to compute1 and have access to a storage allocation.\n", - "2. Use the RIS Knowledge Base and Tutorials to launch a Docker image on a compute1 node.\n", - "3. Drop into the #bfx_workshop slack channel and introduce yourself!\n" + "1. Login to the Google Cloud Console using your WUSTL Key\n", + "2. Drop into the #bfx-workshop slack channel and introduce yourself!\n" ] } ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" },