Multi config testing using git and virtualenv

We often have the problem that we want to test code in multiple configurations, even in the same Python.

For example, for nibabel, we should test on all the supported configurations including:

• numpy only
• numpy and scipy
• numpy and scipy and pydicom

We might also want to test on multiple versions of numpy, scipy, and pydicom.

On all platforms, we will want to test at least on the earliest and latest supported numpy and scipy versions.

For windows and mac, we may well want to test using binary installers to provide the numpy and scipy versions. We may also want to test on 32 bit and 64 bit versions.

On windows we may also want to test against different numpies provided from the numpy installers, Christophe Gohlke's numpy binaries, or EPD.

buildout is the classic tool for this kind of problem. There are some problems with buildout here though. First - nibabel, as other nipy* projects, relies on numpy when running setup.py. nipy itself relies on nibabel as a build-time (setup.py) dependency. These build-time dependencies aren't well handled by buildout and need some nasty hacking to make work. The second problem is that buildout imagines a world where everything is installable via easy_install. This is obviously not the case for binary installers on Mac at least. I haven't yet investigated using msi or exe installers via easy_install.

So, if we don't use buildout, we need to craft a set of virtualenvs the have the dependencies we need. How should we manage these?

Consider an example:

1. make virtualenv from python 2.7 with no site packages
2. install numpy 1.6.2 by hand from source (python setup.py install)
3. make source distribution from nibabel distribution (python setup.py sdist)
4. install into virtualenv (python setup.py install)
5. run tests python -c "import nibabel; nibabel.test()"

We probably want to reuse the work from steps 1 and 2 in our next test run. The virtualenv at the end has nibabel installed and we probably don't want that for the next test run. We don't want to have to trust an uninstall mechanism, either by hand or using some tool (I believe pip does this). So we want to store the virtualenv from step 2 for re-use.

What if we want to make a new virtualenv called np-1.6.2-sp-0.11 which has numpy 1.6.2 and scipy 0.11? We'd like to base that off the np-1.6.2 virtualenv, but I don't think there's a simple mechanism to allow one virtualenv to build on another. So we'd have to install numpy 1.6.2 in a fresh virtualenv, and install scipy on top of that. This will share a great deal of content with the np-1.6.2 virtualenv.