Python type hints and migration to Python 3

Migration to Python 3

See these Google Slides for our plan overview for migrating to Python 3 and adding mypy type hints to help catch problems in the migration and elsewhere (esp. with the string to unicode vs. bytes change).

TODO: Specifically how to use type hints in Python 2 & 3 compatible code to catch problems with bytes vs. unicode strings?

Main References

• style-guide.md#type-hints -- our notes on using Python type hints including numpy stubs.
• Porting Python 2 Code to Python 3.
• What's New in Python.
• The "six" compatibility library.
• Python's "Future" conversion tool.

FYI: Additional References

• Why Python 3 exists
  • Text vs. binary data in Python 2 is an error prone mess. Mixing encoded and unencoded text is unreliable and confusing, e.g. both str and unicode types have .encode() and .decode() methods. Python predates the Unicode standard. Inconsistent unicode handling, e.g. in a script vs. interactive interpreter; also open().read().
  • Python 3 fixes that problem by separating unicode text from binary bytes as separate types. The new scheme is the "Unicode sandwich," "bytes in I/O; unicode in all the app code in between." However, the Python team threw in a lot of other incompatible changes. Big mistake! They won't do that again.
• The Unicode HOWTO.
• Nick Coghlan's Python Notes.
• The Story of Python 2 and 3.
• Ned Batchelder’s Pragmatic Unicode talk/essay
• Supporting Python 3: An in-depth guide.
• differences.
• @twouters’s old TransitionToPython3 wiki.
• What's New in Python for really comprehensive and exhaustive documentation about all language changes since Python 2.7.
• Conservative Python 3 Porting Guide..
• From Dropbox, Incrementally migrating over one million lines of code from Python 2 to Python 3
  • Don't use unicode literals.
  • Use Mypy type checks, unit tests, and Py2 -3 in CI to check for backsliding esp. on unicode/bytes types.
• Python 3 for Scientists.

Strategy

• Adopt all the __future__ imports.
  • Division is the challenging one. It's mostly in use already, with the big exception that wholecell/utils/units.py has truediv turned off for its callers due to Issue #433.
• Adopt Python 3 compatible libraries.
  • The pips should now be Python 3 compatible, but they aren't all clearly marked that way.
  • Finish adopting subprocess32 in place of subprocess. It's a back-port of the Python 3 subprocess with improvements and bug fixes in process launching.
• Incrementally convert to Python 3 compatible syntax and semantics. Use a tool like "future" to do much of the conversion. As we ratchet up the Python 3 compatibility, let everyone know and update the checker tool configuration.
  • Use a checker tool in CI to catch backsliding on Python 3 compatibility changes.
• Add type hints, esp. for the str, bytes, unicode, and basestring types and the AnyStr type hint.
  • Add a type checker in CI, most likely pytest (see below).
• Drop support for Python 2.
• Phase out use of the "six" compatibility library.

Type hints

Type hints look like this:

def emphasize(message):
  # type: (str) -> str
  """Construct an emphatic message."""
  return message + '!'

A few type hints -- esp. one per function definition -- can go a long way to catching problems and documenting types.

PyCharm checks types interactively, while you edit. You don't need any other tools to check types. See Python Type Checking (Guide).

Batch programs mypy and pytest are other ways to check types, particularly in Continuous Integration builds (CI).

Typeshed is a repository for "stub" files that associate type definitions with existing libraries. It's bundled with PyCharm, mypy, and pytype. It does not have types for Numpy.

Types for Numpy

There are experimental type stubs in the numpy repo numpy-stubs that define types for dtype and ndarray. It's not fancy but it does catch some mistakes and it improves PyCharm autocompletion. Hopefully the numpy team will improve these stubs, but numpy is more flexible with types than the type system is unlikely to handle.

With this stub file, you can write type hints like np.ndarray and np.ndarray[int]. It doesn't have a way to express array shape so that still goes into a docstring.

import numpy as np

def f(a):
    # type: (np.ndarray[float]) -> np.ndarray[int]
    return np.asarray(a, dtype=int)

The wcEcoli project includes numpy-stubs.

To install more stub files:

1. Copy them into a stubs/ directory in the project.
2. Mark the stubs/ directory as a source root in PyCharm by choosing Mark Directory as | Sources Root from the directory's context menu.