pkglib's dependency resolution is too strict

[Wilfred]

Scenario 1

• A==1.0 and A==2.0 are available.
• B==1.0 depends on A==1.0.
• C==1.0 depends on A (any version).
• We currently have installed B and A==1.0.

We want to install C. Currently, pkglib will set Resolver.best based on the currently installed packages: https://github.com/ahlmss/pkglib/blob/cf3299a5abdbda5d9a3be535ef96bff74fc11550/pkglib/pkglib/setuptools/buildout.py#L502 . As a result, we'll try to find a version of C that satisfies A==1.0, B==1.0 and everything works fine.

Scenario 2

• A==1.0 and A==2.0 are available.
• B==1.0 depends on A (any version).
• C==1.0 depends on A==1.0
• We currently have installed B and A==2.0.

We want to install C.

This won't work. Because we consider the currently installed packages, pkglib will try to find a version of C that satisfies A==2.0, B==1.0 and we can't proceed.

Scenario 3

• A==1.0 and A==2.0 are available.
• B==1.0 depends on A==1.0
• C==1.0 depends on A==2.0
• We currently have installed B and A==1.0.

We want to install C. This won't work (and indeed it should not).

Solutions

1: Consider everything installed to be a requirement

This is what pkglib currently does, and it fails on scenario 2.

2: Allow anything in the virtualenv to be upgraded as necessary

ahl.pkgutils used to do this (and I ended up restoring this behaviour), and I believe pkglib used to do this too. This fixes scenario 2, but breaks in scenario 1 (B's requirements are ignored, so we end up with A==2.0, B==1.0, C==1.0).

3: Allow a project to declare dependencies.

Ideally, we'd be able to communicate to pkglib that we want both B and C installed, and allow pkglib to find a set of packages that keeps everything happy (so scenarios 1 and 2 succeed) whilst still refusing to proceed in scenario 3.

Status

I'm unable to change our internal code to be based on pkglib, since I've hit scenario 2 and it makes us unable to install some packages. I'd like to change the dependency resolution logic to solution 2 (as it was before) for now.

Thoughts, @eeaston?

[Wilfred]

One interesting side-effect of reverting our dependency resolution logic is that we'd be using the same logic as pip. We could then replace our dependency resolver wholesale with pip's.

[eeaston]

Hi! Interesting case. Sorry that it's causing merge problems - I'll try and make a minimal test case to reproduce. In scenario 3, what do you mean by 'project'? I'm not sure about replacing the resolver with pip, though I haven't dug through pip's code. Do you know if it does backtracking? This and the prefer-final/prefer-dev are the main features of pkglib's resolver.