conda packages only work with conda-forge

[zooba]

I'm trying to add Playwright to my conda environment installed from Anaconda's repository, but it produces conflicts and/or tries to replace the entire environment.

It seems the packages published to anaconda.org/microsoft have tight dependencies to conda-forge, which means it's impossible to install the packages into a non-conda-forge environment.

For example, here's the output of a command that ought to install python=3.10 from Anaconda and playwright from -c microsoft:

(env) C:\Users\...> conda install -c microsoft playwright python=3.10
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
playwright -> typing_extensions -> python[version='>=2.7,<2.8.0a0|>=3.5|>=3.6|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0']
playwright -> python[version='>=3.10,<3.11.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|>=3.11,<3.12.0a0|>=3.7,<3.8.0a0']
python=3.10

(As usual, I don't believe that the conflicts listed are the actual conflicts. The fact that it fails is the main point.)

If I add conda-forge as a channel, I get this proposed plan:

>conda install -c microsoft -c conda-forge playwright python=3.10
...
The following NEW packages will be INSTALLED:

  bzip2              conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
  ca-certificates    conda-forge/win-64::ca-certificates-2023.7.22-h56e8100_0
  greenlet           repo/main/win-64::greenlet-2.0.1-py310hd77b12b_0
  libffi             conda-forge/win-64::libffi-3.4.2-h8ffe710_5
  libsqlite          conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0
  libzlib            conda-forge/win-64::libzlib-1.2.13-hcfcfb64_5
  openssl            conda-forge/win-64::openssl-3.1.4-hcfcfb64_0
  pip                conda-forge/noarch::pip-23.3.1-pyhd8ed1ab_0
  playwright         microsoft/win-64::playwright-v1.33.0-py310_0
  pyee               conda-forge/noarch::pyee-9.0.4-pyhd8ed1ab_0
  python             conda-forge/win-64::python-3.10.13-h4de0772_0_cpython
  python_abi         conda-forge/win-64::python_abi-3.10-4_cp310
  setuptools         conda-forge/noarch::setuptools-68.2.2-pyhd8ed1ab_0
  tk                 conda-forge/win-64::tk-8.6.13-hcfcfb64_0
  typing_extensions  conda-forge/noarch::typing_extensions-4.8.0-pyha770c72_0
  tzdata             conda-forge/noarch::tzdata-2023c-h71feb2d_0
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
  vc                 conda-forge/win-64::vc-14.3-h64f974e_17
  vc14_runtime       conda-forge/win-64::vc14_runtime-14.36.32532-hdcecf7f_17
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.36.32532-h05e6639_17
  wheel              conda-forge/noarch::wheel-0.41.2-pyhd8ed1ab_0
  xz                 conda-forge/win-64::xz-5.2.6-h8d14728_0

However I'm not able to use conda-forge packages in my environment - I have to use Anaconda's (for $place_of_work reasons).

It would probably be best if the anaconda.org/microsoft channel contained packages that work with Anaconda's repository, and then contribute a recipe to conda-forge to make their own build of playwright that works with that channel.

(An alternative would be to ask Anaconda to make their own build of playwright that works with their repository, but they'd want it in conda-forge first anyway, at which point the problem is solved.)

[zooba]

Okay, after a bit more playing, I've managed to get an environment that basically meets my needs.

The problem is the python_abi "package", which doesn't contain any files, but is used to constrain packages (basically, to make conda install fail more often so you're less likely to get a broken environment).

With the following commands, I managed to get an environment that takes python_abi from conda-forge, but has everything else from the correct locations:

> conda install python=3.10
> conda install -c conda-forge python_abi
> conda install -c microsoft playwright
> conda list
# packages in environment at C:\Users\...\envs\env:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                he774522_0
ca-certificates           2023.08.22           haa95532_0
greenlet                  2.0.1           py310hd77b12b_0
libffi                    3.4.4                hd77b12b_0
openssl                   3.0.11               h2bbff1b_2
pip                       23.3            py310haa95532_0
playwright                v1.33.0                 py310_0    microsoft
pyee                      9.0.4           py310haa95532_0
python                    3.10.13              he1021f5_0
python_abi                3.10                    2_cp310    conda-forge
setuptools                68.0.0          py310haa95532_0
sqlite                    3.41.2               h2bbff1b_0
tk                        8.6.12               h2bbff1b_0
typing_extensions         4.7.1           py310haa95532_0
tzdata                    2023c                h04d1e81_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.41.2          py310haa95532_0
xz                        5.4.2                h8cc25b3_0
zlib                      1.2.13               h8cc25b3_0

The three commands need to be run separately, otherwise they'll each try to replace most of the environment. (The conda-forge step actually downgrades my TLS certificates to an older version, but the third step brings them back.)

So there's a workaround for users. I'm not sure there's an easy way to avoid the python_abi dependency though, other than to compile using Anaconda packages instead of conda-forge. Clearly all the other dependencies are available.

[mxschmitt]

I was looking at some other packages which we distribute over our microsoft conda channel, their Python dependency looks similar, and it seems like there is not much we can do here on our side? Where is our python_abi dependency coming from? Most likely from Python itself?

Seems like you are more of a Conda expert here than we are here, so if you have suggestions, let us know!

[zooba]

I was looking at some other packages which we distribute over our microsoft conda channel, their Python dependency looks similar

I never accused you of being the only culprits 😉

Where is our python_abi dependency coming from?

It comes from using conda-forge when building the package, because they make it a strict requirement. Anaconda does not, so if you build using their packages, it would still successfully install against conda-forge because all the dependencies are satisfied. It doesn't work the other way, because Anaconda can't satisfy the python_abi dependency.

if you have suggestions, let us know

I suspect deleting this line will be sufficient: https://github.com/microsoft/playwright-python/blob/main/.github/workflows/publish.yml#L19

Possibly it needs to be replaced with defaults, but that should be the default.

[mxschmitt]

I suspect deleting this line will be sufficient: https://github.com/microsoft/playwright-python/blob/main/.github/workflows/publish.yml#L19

Possibly it needs to be replaced with defaults, but that should be the default.

I tried both, but getting this in both casese:

Unsatisfiable dependencies for platform osx-64: {'greenlet==3.0.0', 'pyee==11.0.1'}

[zooba]

Looks like Anaconda hasn't updated past greenlet 2.0.1 and pyee 9.0.4 for osx-x64 (list here, but be warned it's a big page).

Are those particular versions required? Or just frozen there. We can ask them to prioritise the newer versions if needed and get them added.

[mxschmitt]

We usually freeze them, so that if customers install Playwright, its guaranteed that this version works for them. We had cases where dependencies were breaking us.

We are also using very often very recent versions of pyee or greenlet, since they often fix bugs which are important for our customers.

So if we would lax the version range for our published conda package, it would work with the default channel? TIL.

[zooba]

That sounds like a good reason, but as I said, we can ask Anaconda to accelerate their updates if there are fixes we need.

And yeah, loosening version ranges is generally a good move. Conda does a pretty good job of locking in the versions that are actually used to build a package if they matter, and allowing updates on install for those that don't. So you should be able to get the best of all worlds by not freezing the inputs and making sure that the outputs are/aren't frozen as appropriate.

Happy to take this internal to figure out how to proceed from here, you know my team alias (the obvious one).