py2py3-talk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Python is dead, long live Python!\n",
    "### Adapting old codebase for Python 3.x compatibility - case study\n",
    "\n",
    "Łukasz Rogalski"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## About me\n",
    "\n",
    "![title](photo.png)\n",
    "\n",
    "- rogalski.91 /at/ gmail /dot/ com\n",
    "- https://github.com/rogalski\n",
    "- https://www.linkedin.com/in/lukasz-rogalski"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What to expect from this talk?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- infrastructure "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- philosophy\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- good starting point rather than *extensive* overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- (hopefully) spark for fruitful disscussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Agenda\n",
    "1. Motivations for migration\n",
    "2. Case study\n",
    "3. Key learnings\n",
    "4. Q&A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why migrate?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### New features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- stdlib improvements\n",
    "    - `enum`s\n",
    "    - `pathlib`\n",
    "    - `dataclass`es\n",
    "    - `subprocess` module with timeouts\n",
    "    - `asyncio` \n",
    "    - ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- syntax improvements\n",
    "    - type hints\n",
    "    - native `async` / `await` support\n",
    "    - extended iterable unpacking\n",
    "    - keyword-only arguments\n",
    "    - ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why migrate?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Python 2.x deprecation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[PEP 373 -- Python 2.7 Release Schedule](https://www.python.org/dev/peps/pep-0373):\n",
    "> Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ultimate goal - to make your live easier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Case study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## About project\n",
    "- test automation framework\n",
    "- CLI for running tests on local machine\n",
    "- mostly used as an element of bigger fully-automated environments\n",
    "- also used by individuals \"on their desk\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Codebase & infrastructure\n",
    "- quite old (started in 2012), Python 2.6 support at some point\n",
    "- some unit-level tests (but far from 100% coverage)\n",
    "- trunk based development, short-lived feature branches via Gerrit\n",
    "- continuous integration, auto deploy to staging env, manual deploy to prod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem statement\n",
    "### We cannot stop supporting Python 2.x without any notice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Prepare users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- select a date and stick to it\n",
    "- add deprecation info to official docs\n",
    "- send (multiple) e-mails within organization with announcement of pending deprecation\n",
    "- extensive `log.warning` for CLI invocations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Make codebase Python 3.x compatible"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "But keep backwards compatibility with 2.x interpreter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What backwards compatibility means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**How clients are using your code?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- if you do web dev and deploy Docker images: no one cares about Python version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- if you develop library: importable API needs to be kept intact"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- if you develop CLI tool: old command line options / settings files etc. needs to be supported"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Before migration - few words about backports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **backport** - stdlib package available for older Python versions\n",
    "- maintained on PyPI by volunteers\n",
    "- easy way to use *some of* new features\n",
    "- installation criteria controllable from your app requirements, e.g. `singledispatch==5.4; python_version < '3.3'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can avoid messy import statements by preparing compatibility module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Backports - example 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# compat.py\n",
    "import sys\n",
    "if sys.version_info < (3, 3):\n",
    "    from singledispatch import singledispatch\n",
    "else:\n",
    "    from functools import singledispatch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# code using singledispatch\n",
    "from .compat import singledispatch\n",
    "\n",
    "@singledispatch\n",
    "def fun(arg):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Backports - example 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# compat.py\n",
    "import sys\n",
    "if sys.version_info < (3, 0):\n",
    "    import subprocess32 as subprocess\n",
    "else:\n",
    "    import subprocess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# code using subprocess or subprocess32\n",
    "from .compat import subprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Backports - summary\n",
    "- enables usage of new features with older intepreter\n",
    "- use them if needed\n",
    "- good *meantime* solution until migration is fully completed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let's migrate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**First idea** - let's do this automatically!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`2to3` (stdlib) is one-way code converter from Py 2.x. to Py 3.x.\n",
    "\n",
    "*Backwards compatibility needs to be kept (at least for now).*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`python-modernize` is code converted to apply fixes while keeping Py 2.x compatibility.\n",
    "\n",
    "*After some trial runs we decided not to use it.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`pylint --py3k` is a Pylint's special checker mode focused on porting code.\n",
    "\n",
    "*Pre-existing `pylint` configuration extended to use `--py3k` checkers as well.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let's migrate\n",
    "Second idea -  **manual** approach (with some help of existing libraries)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Starting point - what about unit tests?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- start running unit tests on Python 3.x\n",
    "- `tox` used to manage virtual environments\n",
    "- reproducible test environment for Windows / Linux / macOS\n",
    "- easy to start, just run `tox -e py36`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Ideally, failing tests will uncover all issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There are libraries that can help with this process of fixing code.\n",
    "- `six`\n",
    "- `python-future`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Laziness of collections methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- `dict.iterkeys()` vs. `dict.keys()`\n",
    "- `xrange()` vs. `range()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can use `six` to achieve consistent behavior in both Python 2.x and Python 3.x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import six\n",
    "d = {\"a\": 1, \"b\": 2}\n",
    "for k, v in six.iteritems(d):\n",
    "    print(k, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Our collections were rather small. We decided to use newer syntax only at the cost of being suboptimal on Python 2.x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "d = {\"a\": 1, \"b\": 2}\n",
    "for k, v in d.items():\n",
    "    print(k, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Module/object renames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Use `six`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from six.moves import winreg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Or do error handling manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import winreg\n",
    "except ImportError:\n",
    "    import _winreg as winreg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More drastic stdlib reorganizations - `urllib`, `urllib2`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Significantly redesigned in Python 3.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Use `six`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from six.moves.urllib.parse import urllib\n",
    "from six.moves.urllib.error import HTTPError\n",
    "from six.moves.urllib.request import urlopen\n",
    "from six.moves.urllib.response import addinfo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Or don't use those libraries at all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In our project `urllib`-dependent code was rewritten to use `requests`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `__dunder__` methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Some protocols / operators changed their magic method name in Python 3.x.\n",
    "- `__nonzero__` became `__bool__` (truthiness and falsiness)\n",
    "- `next` became `__next__` (iterator protocol)\n",
    "- ...\n",
    "\n",
    "There is a neat trick to handle this on code level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class MyClass(object):\n",
    "    def __bool__(self):  # python 3.x\n",
    "        return self.member != 0\n",
    "    \n",
    "    __nonzero__ = __bool__  # python 2.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `__future__` imports\n",
    "\n",
    "> A future statement is a directive to the compiler that a particular module should be compiled using syntax or semantics that will be available in a specified future release of Python where the feature becomes standard.\n",
    "\n",
    "Enabling `print_function`, `absolute_import` and `division` - highly recommended.\n",
    "\n",
    "Enabling `unicode_literals` can introduce some cryptic errors and [opinions on using it varies](https://python-future.org/unicode_literals.html#should-i-import-unicode-literals).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Are your unit tests good? (mock curse - example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We have bunch of code with interacts with subprocesses.\n",
    "\n",
    "Launch a process, parse output, return accordingly to output and/or return code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import subprocess\n",
    "\n",
    "\n",
    "def my_func():\n",
    "    output = subprocess.check_output(\"ping -c 1 127.0.0.1\", shell=True)\n",
    "    return 'some_string' in output\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    my_func()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import mock\n",
    "\n",
    "import my_module\n",
    "\n",
    "\n",
    "def test_fn():\n",
    "    mocked_output = 'my_mocked_output'\n",
    "    with mock.patch(\"subprocess.check_output\", return_value=mocked_output):\n",
    "        assert not my_module.my_func()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Everything is good, right?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Unit test is valid on Python 2.x."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Unit test is passing on both Python 2.x and Python 3.x."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Code works on production on Python 2.x."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Code **fails** on production on Python 3.x\n",
    "\n",
    "```\n",
    "Traceback (most recent call last):\n",
    "  File \"my_module.py\", line 10, in <module>\n",
    "    my_func()\n",
    "  File \"my_module.py\", line 6, in my_func\n",
    "    return 'some_string' in output\n",
    "TypeError: a bytes-like object is required, not 'str'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why?\n",
    "\n",
    "- Python 2.x - concept of bytes and strings was ambiguous\n",
    "- Python 3.x - explicitly discriminates between bytes and unicode strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Be aware of quirks like that!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## About unit tests and coverage\n",
    "- `coverage` library can be used to collect data about unit test coverage info\n",
    "- now you have two sets of data (one for Python 2.x, other for Python 3.x)\n",
    "- luckily, it is possible to merge info from multiple environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "tox.ini\n",
    "```\n",
    "[env]\n",
    "setenv=COVERAGE_FILE=.coverage.{envname}\n",
    "\n",
    "```\n",
    "\n",
    "Postprocessing:\n",
    "```\n",
    "coverage combine\n",
    "coverage html\n",
    "```\n",
    "See [official docs](https://coverage.readthedocs.io/en/latest/cmd.html#cmd-combining) for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Unit tests passing locally 🎉 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- configure CI server to run UT both interpreters\n",
    "- extremely easy (only dependencies: `python3` and `tox` available on build agent)\n",
    "- `tox` takes care of creating reproducible environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In perfect world, you are done here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In practice, not really."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "App failed miserably even though all of unit tests were passing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Local integration testing\n",
    "**Basic idea**:\n",
    "1. Prepare environment with all dependencies (`tox` takes care of that)\n",
    "2. Set up command line, settings files etc.\n",
    "3. Run app\n",
    "4. Check whether passing criteria occurred:\n",
    "  - exit code = 0\n",
    "  - no errors logged\n",
    "  - output files with specific content created\n",
    "  - etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How to implement that?\n",
    "Set up extra environments in `tox.ini` file!\n",
    "```\n",
    "[testenv]\n",
    "deps= -rrequirements.txt\n",
    "commands= pytest --cov=my_package {posargs:tests/unit/}\n",
    "\n",
    "[env:py27-integration]\n",
    "basepython = python2.7\n",
    "commands = pytest {posargs:tests/integration/}\n",
    "\n",
    "[env:py36-integration]\n",
    "basepython = python3.6\n",
    "commands = pytest {posargs:tests/integration/}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Running those is as simple as:\n",
    "- `tox -e py27-integration`\n",
    "- `tox -e py36-integration`\n",
    "\n",
    "What you get?\n",
    "- Much slower than unit tests, but...\n",
    "- Quick way for developer for running sanity integration tests\n",
    "\n",
    "**Make those tests pass!**\n",
    "\n",
    "We eventually added those tests as step in CI server anyway."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Start using Python 3 in staging environments\n",
    "Release pipeline:\n",
    "\n",
    "Checkin -> CI Server (UT, integration tests) -> Staging environment -> Manual deploy to production"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Started to use Python 3.x in staging environment:\n",
    "- Both Python 2.x and Python 3.x are tested (same as in UT and integration tests)\n",
    "- **Make those tests pass**\n",
    "- Monitor for regressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Phased rollout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- identify independent areas where we can start using Python 3.x in production"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- enable those one by one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- if something got messed up, damage is more manageable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## First non-Py2.x compatible release"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Increment a major number of app versioning scheme**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Develop first functionality using Python 3.x features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Clean up code for supporting Python 2.x.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary - key learnings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- give (reasonably long) ETA for compatibility breakage to your users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- use backports if needed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- be aware of differences between 2.x and 3.x branches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- do not treat migration as purely mechanical operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- automate your testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- take a calculated risk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Thank you!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Contact\n",
    "- rogalski.91 /at/ gmail /dot/ com\n",
    "- https://github.com/rogalski\n",
    "- https://www.linkedin.com/in/lukasz-rogalski"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Q&A"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}