Refreshing lockfiles and groups

[adriangb]

As @frostming pointed out on Twitter Pydantic is now using PDM for package/dependency management.
I've struggled to get groups and lockfile refreshing to work correctly.

We have what I think is not too crazy of a setup: a docs group, a test group, etc and a dependency (pydantic-core) that we want to update pretty often without updating everything else. If you check out our main branch (say pydantic/pydantic@f93966d) update pydantic-core in pyproject.toml (in this case from ==0.38.0 to ==0.39.0) and run pdm lock --refresh --dev --group :all or pdm lock --refresh or any other variation I can think of it locks successfully but then when I try pdm install --group :all I get:

[ResolutionImpossible]: [RequirementInformation(requirement=NamedRequirement(name='pydantic-core', marker=None, extras=set(), 
specifier=<SpecifierSet('==0.39.0')>, editable=False, prerelease=False), parent=None)]

This doesn't make much sense because (1) it locked successfully so I'm not even sure why it's resolving, if anything it should just fail to download and (2) 0.39.0 is on PyPi and pip install works.

What am I doing wrong?

I'll also point out that in the case of resolution failure Poetry gives a very nice error message explaining why the resolution failed (xyz>=1.2.0 depends on abc < 1.0 and yourpackage depends on abc > 1.2, etc.). I'm not sure if it's applicable in this situation (again, I feel like resolution shouldn't even be happening here) but if it is it would be nice to get an error message like Poetry's since the current one is not very actionable.

[frostming]

You need to update the pins in the lockfile but pdm lock --refresh doesn't do that, which is said in the help message: Don't update pinned versions, only refresh the lock file. Though it might be a confusing of what it is updating, it's indeed only updating the hashes(including lockfile hash and artifact hashes).

Just leave the --refresh, run pdm update and it will work. Don't worry about the unnecessary IO, it will check the current pins and only request PyPI to get new when it doesn't meet the new requirements.

[adriangb]

That does seem closer to what we want, thank you. I do think it's confusing that the docs say Update package(s) in pyproject.toml for pdm update which is definitely not what I want: I want to update pdm.lock with the minimum amount of changes needed to have it in sync with pyproject.toml. In other words, I want poetry lock --no-update.

What seems to not be working now is dependency groups. With poetry when you do poetry lock it automatically includes all of the dependency groups in the lockfile so that if then you want to do something like poetry install -G testing it just works. I don't see any cons to this behavior, but it seems that pdm requires me to explicitly list all dependency groups e.g. pdm update --group testing --group docs .... I tried pdm update --group :all but it doesn't seem to do what I want. Any advice here?

[pawamoy]

Users have been asking to be able to lock groups independently. You can lock into a custom lock file, and install from a custom lock file.

If you have no need for such a feature (like me), I recommend locking with -G:all indeed, so that you can install any group(s) with pdm install from the same, default lock file.

I never use the update or sync commands so can't recommend anything here.

[adriangb]

Thanks. Using —group :all did seem to work after all. Not sure if I was doing something wrong or a delete and re-create of the lockfile was necessary. Nonetheless, I do feel like making that the default and opting into locking groups into separate lockfiles would make more sense given that (1) having incompatible dependencies (the only reason I can think of for multiple lockfiles) seems like an advanced use case (2) this is the default behavior for poetry and Cargo.

[pawamoy]

a delete and re-create of the lockfile was necessary.

I believe it is necessary if you first locked without all groups. I had some trouble as well. It's kind of a reflex now when I want to update all deps or when something goes wrong: I just delete pdm.lock and relock everything.

I do feel like making that the default [...] would make more sense

I agree. Note that it's mostly the case though: by default PDM locks everything (prod+dev) except optional groups (correct me if I'm wrong @frostming), so if you don't have optional groups, pdm lock == pdm lock -G:all is true.