False Environment Marker in dependency triggers error "[InvalidPyVersion]: Invalid specifier: 'impossible'"

[poirier]

• [ x] I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

Install PDM 2.7.4. I have it in ~/.local/bin.

In an empty directory, create pyproject.toml:

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"
[project]
name = "strange"
dependencies = [
    "backports.zoneinfo;python_version<\"3.9\"",
]
requires-python = ">=3.10"

Use Python 3.10.*. In my case, .pdm-python contains /run/current-system/sw/bin/python3.10.

Run pdm update -v

Actual behavior

Output is:

$ pdm update -v
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   backports-zoneinfo; python_version < "3.9"
pdm.termui:   python>=3.10
pdm.termui:   Adding requirement backports-zoneinfo; python_version < "3.9"
pdm.termui:   Adding requirement python>=3.10
pdm.termui: ======== Starting round 0 ========
[InvalidPyVersion]: Invalid specifier: 'impossible'

Expected behavior

Since Python version is 3.10, the "backports-zoneinfo" requirement should be ignored. See https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
PDM version:
  2.7.4
Python Interpreter:
  /nix/store/rxwdzwr60hflvy4lnkiqhkjrmj9q0pwx-python3-3.10.11-env/bin/python3.10 (3.10)
Project Root:
  /home/poirier/src/test_pdm
Local Packages:
  /home/poirier/src/test_pdm/__pypackages__/3.10
{
  "implementation_name": "cpython",
  "implementation_version": "3.10.11",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "6.3.5",
  "platform_system": "Linux",
  "platform_version": "#1-NixOS SMP PREEMPT_DYNAMIC Tue May 30 13:17:29 UTC 2023",
  "python_full_version": "3.10.11",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "linux"
}

[pawamoy]

I think your dependency marker and your required Python version are incompatible, and that's why PDM cannot resolve dependencies. The currently selected Python version has no incidence, only the metadata in pyproject.toml.

[poirier]

The idea of the environment markers is to be able to say, if I'm running this with Python before 3.9, I need this requirement; otherwise, I don't. It helps make packages compatible with more versions of Python. So in the case that we're running Python 3.10, a build tool can just ignore this dependency.

[pawamoy]

Yes I know what markers are for 🙂 In your case though, your required Python version is incompatible with your dependency marker. Your project can never be installed on Python 3.8 because you specify requires-python = ">=3.10". So the marker makes no sense here, and the whole dependency can simply be dropped.

[poirier]

Though I guess if I'm trying to make this compatible with Python < 3.9, I shouldn't be saying I require Python >= 3.10, should I? I can try taking it out.

Still, given how environment markers work, I don't think this should fail, since I am meeting the required Python version and the requirement is tagged as not needed for the Python version I'm using.

[pawamoy]

Or yes, you can change to requires-python = ">=3.8" (or lower, but not higher, since your marker says <3.9).

[poirier]

Getting rid of the environment markers or lowering my required Python version makes the error not happen. So this isn't a big problem for me.

I do think, though, that the spirit of environment markers isn't implemented quite right here - from the link I cited before, "A marker expression evaluates to either True or False. When it evaluates to False, the dependency specification should be ignored.", and here the dependency specification is not being ignored.

[pawamoy]

I think PDM correctly handles markers. Just like dependencies must be compatible between them, the required Python version and the dependencies must be compatible. @frostming would probably explain it better though 🙂 I'll move this to a discussion!

[frostming]

when doing cross-platform locking, the environment markers are not evaluated and no deps will be dropped at lock time.

But python requires is an exception. Subdependencies specified with a disjoint python range will be ignored, but top deps won't because we assume users know what they want. We may need to improve this.