Homebrew and Python

Python is a powerful and beautiful interpreted language.

Overview

This page describes how Python is handled in Homebrew and how to write formulae that install Python bindings.

Homebrew should work with any CPython, in particular with OS X system's Python. Further, Homebrew provides formulae to brew a more up-to-date Python 2.7.x. (and 3.x). However, if you choose to use another Python than these two alternatives (system Python or brewed Python), the Homebrew team can only provide limited support.

Brew Python

We recommend to brew install python because:

• It is newer
• Comes with pip (and setuptools)
• Uses readline (better than libedit)
• Sqlite with loadable extensions
• Use newer OpenSSL (optionally)
• Use newer Tcl/Tk (optionally)
• Python (distutils) finds brewed software (includes, libs), knows about the compiler and flags even if the command line tools for Xcode are not installed.
• No need to set the PYTHONPATH for Homebrew bindings.
• No need to work-around the sudo-is-needed-for-easy_install issue, described there --> Gems, Eggs and Perl Modules.

Python 2.x or Python 3.x

Homebrew provides a formula for Python 2.7.x and one for Python 3.x. They don't conflict, so they can both be installed. The executable python will always point to the 2.x and python3 to the 3.x version. But which version should I use?

Since June 2013, brew supports to build bindings for both major Python versions, 2.x and 3.x, however each formula has to be adapted and tested (some project just don't yet provide 3.x support). As of June 2013, formulae --with-python3 support are Sip, PyQt, Shiboken, PySide and Swig. More to come. Note, Python 2.x bindings are installed, too, so you get both, unless you say --without-python.

Your help on adding 3.x support for formulae is very welcome!

Setuptools, Pip, etc.

The Python formula installs pip and Setuptools.

Setuptools can be updated via Pip, without having to re-brew Python:

pip install --upgrade setuptools

Similarly, Pip can be used to upgrade itself via:

pip install --upgrade pip

Note, pip install --user is disabled for brewed Python. This is a bug in distutils, because Homebrew writes a distutils.cfg that sets the prefix for packages. A possible work-a-round (that will put executable scripts in ~/Library/Python/<X>.<Y>/bin) is:

pip install --user --install-option="--prefix="

The site-packages and the PYTHONPATH.

The site-packages is a directory to contain Python modules, especially bindings installed by other formulae. Homebrew creates such a directory at

$(brew --prefix)/lib/pythonX.Y/site-packages`

for example /usr/local/lib/python2.7/site-packages for python2.7. The reasoning is that for (minor) upgrades or reinstalls of Python, your modules are not lost. And a rather strict Homebrew policy is not to write stuff outside of the brew --prefix, so we don't spam your system.

A brewed Python 2.7 also searches for modules in:

• /Library/Python/2.7/site-packages
• ~/Library/Python/2.7/lib/python/site-packages

Homebrew's site-packages directory is first created if any Homebrew formula with Python bindings is installed or if brew install python. A brewed Python already knows about this dir.
For other Pythons you'll have to add that dir to your PYTHONPATH environment variable. You may want to append a line like so to your hidden configuration file .bash_profile in your home dir:

This is not needed if you use a brewed Python!

touch ~/.bash_profile
echo export PYTHONPATH=\"$(brew --prefix)/lib/python2.7/site-packages:\$PYTHONPATH\" >> ~/.bash_profile
source ~/.bash_profile
echo $PYTHONPATH

Homebrew provided Python bindings

Some formulae provide python bindings. Sometimes a --with-python or --with-python3 option has to be passed to brew install in order to build the python bindings. Check with brew options <formula>.
If you have a brewed python, then the bindings are installed for that one. But if you don't have a brewed Python, Homebrew basically just uses the first python (and python-config) in your PATH. Check that by which python.

Warning, Python may crash (see Common Issues) if you import <module> in a different python interpreter than the one that was used during the brew install <formula_with_python_bindings>. Therefore, if you decide to switch between a brewed and system python, then re-‍install all formulae that provide python bindings (such as pyside, wxwidgets, pygtk, pygobject, opencv, vtk, boost to name just a few).

Other - non-homebrew - Python bindings

Our policy is that these should be installed via pip install <x>. To discover, you can use pip search, the new http://crate.io or http://pypi.python.org/pypi. Note, system Python does not provide pip but you can easy_install pip to fix that.

For a brewed Python, modules installed by pip or with python setup.py install, will be put into the before-mentioned $(brew --prefix)/lib/pythonX.Y/site-packages directory, too. Executable python scripts will be in $(brew --prefix)/bin. (To behave more standard conform, brew no longer puts Python scripts into share/python/$(brew --prefix))

Further, a brewed Python (or more precisely distutils) knows which compiler flags to set in order to build bindings for software installed in Homebrew (Find the includes, the libs, the compiler and so forth). In contrast to that, system's Python does not know about the the correct flags, so you may need to

CFLAGS=-I$(brew --prefix)/include LDFLAGS=-L$(brew --prefix)/lib pip install <package>.

Virtualenv

A brewed Python works nicely with Virtualenv. However, when you brew install formulae that provide Python bindings, you should NOT be in an active virtual environment. Activate the virtualenv after you have brewed or, alternatively, brew in a fresh Terminal window. Homebrew will still install Python modules into Homebrew's site-packages and not into the virtual environment's site-package. Virtualenv has a switch to allow "global", i.e. Homebrew's, site-packages to be accessible from within the virtualenv.

How to create a formula with nice Python bindings

Just add the following :special dependency to the formula:

depends_on :python

This assures that Homebrew looks for a suiteable CPython 2.6+ and sets up the PATH accordingly (among a few other things described further down). Not declaring this line typically leads to error messages like "Python.h not found".

You may want to let the user opt-out via --without-python:

depends_on :python => :recommended

or to opt-in (here for 3.x) via --with-python3:

depends_on :python3 => :optional

These options are automatically generated. In the formula you can check, as usual, via build.with? 'python' what the user has decided. To depend on a specific version, instead use

depends_on :python => "2.7"

If you need to specify that not only Python itself but also certain modules are needed:

depends_on :python => ['numpy', 'docutils']
depends_on :python => ['enchant' => 'pyenchant']

If the software provides a setup.py

Usually this line will do the trick

system python, "setup.py", "--prefix=#{prefix}"

Note that python is not a string here because inside of a formula it is an object that points to the correct python binary and offers some support methods, too. More on this later.

The --prefix=#{prefix} part is to achieve that the Python bindings are installed into the Cellar for that specific formula in:

$(brew --prefix)/Cellar/<formula>/<version>/lib/python2.7/site-packages

When brew link is run (automatically at the end of brew install), the Python modules should be linked into $(brew --prefix)/lib/pythonX.Y/site-packages and the scripts should go into $(brew --prefix)/bin. This enables brew to unlink/link/uninstall cleanly.

If the setup.py is older, it may need two additional arguments to avoid writing an easy-install.pth file which will conflict with the easy-install.pth already installed by pip/setuptools. So if you get a brew link problem mentioning this file, you'll need to add to the setup.py args:

"--single-version-externally-managed", "--record=installed.txt"

If the formula uses cmake

Unfortunately, Cmake hard-codes the System Python even if another python is in the PATH. Therefore, you'll have to set something like this (read about the python do block down below):

def install
  ...
  python do
    # Cmake picks up system Python on OS X
    args << "-DPYTHON_INCLUDE_DIR='#{python.incdir}'"
    args << "-DPYTHON_LIBRARY='#{python.libdir}/lib#{python.xy}.dylib'"
    # Set the prefix for the python bindings to the Cellar
    args << "-DPYTHON_SETUP_ARGS:STRING='--prefix=#{prefix} --single-version-externally-managed --record=installed.txt'"
    system 'cmake', '.', *args
    system 'make'
    system 'make install'
  end
end

You may want to have a look at the homebrew/science/vtk.rb formula.

If the formula uses configure/make

Mostly the ./configure provided by the software we want to install is able to find python or python-config and/or look at the PYTHON var. Both are set up by Homebrew during brewing. Often a --with-python or similar flag can be given to configure. Check with ./configure --help. If the configure and make scripts do not want to install into the Cellar, one option is to first call ./configure --without-python (or a similar named option) and then cd into the directory containing the Python bindings and calling the setup.py explicitly as described above. Sometimes we have to inreplace a Makefile to use our prefix for the python bindings. (inreplace is a helper method of Homebrew to grep and edit text files on-the-fly).

The python object contains support methods

In the Formula class there is a python object, that supports some convenience methods which do the complicated work of figuring out the information no matter which python version or brewed vs. python from OS X:

python.site_packages # the site-packages in the Cellar
python.global_site_packages # homebrew's global one
python.binary # the full path to the python binary.
python.prefix
python.version
python.version.major
python.version.minor
python.xy # => e.g. "python2.7"
python.incdir # includes of python
python.libdir # the python dylib library
python.framework # The dir that contains the Python.framework
python.pkg_config_path # used internally by brew
python.from_osx?
python.framework?
python.universal?
python.pypy?
python.standard_caveats # The usual text to set PYTHONPATH that is only shown for python.from_osx?
python.if3then3 # => "" for 2.x and to "3" for 3.x.

Instead of using python, there is also python2 and python3 to explicitly refer to 2.x and 3.x resp..

To play around, type brew irb and enter (for example):

PythonInstalled.new.prefix

Additionally the python, python2 and python3 objects return false if Python 2.x (or 3.x respectively ) is not available.

if python2
  # do something specific for python 2.x
  # Of course you can also use `if build.with 'python'`
end

if python3
  # do something specific for python 3.x
end

unless python
  # do something if there is no python (or python support was disabled)
end

Technical details

Formula authors necessarily don't need to read this.

Adding depends_on :python triggers the following actions:

• It is checked if a brewed Python with the minimal required version is available and executable. If so, that one will be used and $(brew --prefix)/opt/python/bin will be inserted into the PATH. A pypy or a Python 3.x is not sufficient. Symlinking python to a Python 3.x will raise an error.
• If no brewed python, the user PATH (the original PATH, not the superenv PATH) is searched for a suitable python executable (python3 for 3.x).
• $(brew --prefix)/lib/pythonX.Y/site-packes/sitecustomize.py is written with certain fixes:
  • Allow other non-brewed python to parse .pth files
  • filter out dirs from the PYTHONPATH, starting with /System/...
  • remove the hard coded site-package location in the Cellar (where we installed Python to) so that pip uninstall works.
• A distutils.cfg is written to define the prefix to be the brew --prefix for user installed python modules. It also makes the builds verbose, so we can more easily debug brewing python bindings.
• The PYTHONPATH is set (internally only), so non-brewed Python do find brewed python modules.
• PYTHONHOME is reset.
• PYTHONNOUSERSITE is defined to avoid mixing user packages with brewed ones.
• CMAKE_INCLUDE_PATH is set to include the Python's incdir.
• PKG_CONFIG_PATH is set to include the python.pc file.