[Feature]: Automatically upgrade built-in package when dependent requires newer version

[amano-kenji]

Confirmation

• I have checked the documentation (README, Wiki, docstrings, etc)
• I am checking these without reading them.
• I have searched previous issues to see if my question is a duplicate.

Elpaca Version

Elpaca 4e68fdf

Operating System

Gentoo Linux

Description

I know this issue was closed previously, but it is still not fixed.

I tried elpaca today, and discovered that I could not install magit because elpaca doesn't install a newer version of seq from elpa, melpa, or any other repository because it is already installed as a built-in package.

The fact that magit cannot be installed by default is an issue.

It seems to me that elpaca should not care which repository a package comes from by default.

If a newer version of seq is required and is only available from elpa or melpa, it should be installed.

Restricting the source of a package to a specific repository should be something users explicitly choose to do.

What about preferring built-in packages by default, but installing third party packages if required?

[progfolio]

Elpaca Version

Elpaca 4e68fdf

Please use the output of M-x elpaca-version in future issues.
It includes additional information, such as your emacs-version.

I tried elpaca today, and discovered that I could not install magit because
elpaca doesn't install a newer version of seq from elpa, melpa, or any other
repository because it is already installed as a built-in package.

You can install a newer version of seq by adding a declaration for it. e.g.

(elpaca seq)

You'll also want to see #270, as seq is a pre-loaded, core package.

The fact that magit cannot be installed by default is an issue.

Magit can be installed. It's just that the Emacs version you're using did not satisfy the latest version of magit's minimum requirement for seq. So you'll either need to update seq as above, or install a version of magit compatible with the built-in version of seq for your Emacs. e.g.

(elpaca (magit :tag "v3.3.0"))

It seems to me that elpaca should not care which repository a package comes from
by default.

I'm not sure what you mean.
It uses the information from the recipe to determine where to download the package repository.
So it doesn't "care", so long as the recipe points to a valid URL for a git repository.

Restricting the source of a package to a specific repository should be something
users explicitly choose to do.

That's currently how it works. See the documentation for the :inherit recipe keyword as well as the manual section on recipe inheritance.

If a newer version of [a built-in library] is required and is only available
from [an ELPA], it should be installed.

I'm not convinced that should be the default behavior.
Consider the case where one package requires a bleeding-edge version of a built-in package, but installing that new built-in package breaks several other packages which do not rely on the latest version.
The user will have to make a decision no matter the default.
Perhaps a user option is in order, but I'll have to think about it more.

[amano-kenji]

I didn't have this issue with package.el. Perhaps, elpaca should automatically install an older version of magit that's compatible with the built-in seq?

Automatically installing an older version of magit is less problematic than updating the built-in seq.

[progfolio]

I didn't have this issue with package.el.

package.el recently added the package-install-upgrade-built-in user option.
Perhaps I'll add something similar in the future.

Perhaps, elpaca should automatically
install an older version of magit that's compatible with the built-in seq?

Again, that can be specified in the declaration. e.g. (elpaca (magit tag: "v3.3.0"))

Automatically installing an older version of magit is less problematic than
updating the built-in seq.

If that works better for you, then I suggest doing that via the means I've shown in this thread for now. Elpaca ships with nearly 6K recipes, but they're not going to be perfect for every situation. In those cases, users should alter the recipe to meet their needs.

[brabalan]

For the record, this is how I installed magit following the advice I found in the issues

(defun +elpaca-unload-seq (e)
  (and (featurep 'seq) (unload-feature 'seq t))
  (elpaca--continue-build e))

(defun +elpaca-seq-build-steps ()
  (append (butlast (if (file-exists-p (expand-file-name "seq" elpaca-builds-directory))
                       elpaca--pre-built-steps elpaca-build-steps))
          (list '+elpaca-unload-seq 'elpaca--activate-package)))

(use-package seq
  :ensure `(seq :build ,(+elpaca-seq-build-steps)))
(use-package transient :ensure t)
(use-package magit :ensure t)

but then I decided to be reasonable and I started over with simply

(use-package magit :ensure (:tag "v3.3.0"))

[meedstrom]

I agree with OP, I was about to open the exact same issue.

My initfiles have this, which corrects the problem, but I worry about users who do not:

(setq elpaca-ignored-dependencies
      (delq 'transient elpaca-ignored-dependencies))

because more and more packages now depend on Transient features that come after the 0.4.3 version that came with Emacs 29. This one just for example melpa/melpa#9131.

It breaks the flow where you are supposed to be able to install a package and pull in all the dependencies that package needs. If a pacakge literally lists (transient "0.7.4") in its Package-Requires, it seems to me a bug if a package manager does not react to that.

I understand this info is not in the MELPA recipe, so not sure how I would fix that other than inspecting all source files...

Anyway, the problem will have to fall into someone's lap, the question is just whose. As a packager, if there are package managers out there that won't install the Transient I declare as minimum, I have no choice but to either stick with Transient 0.4 features or drop support for Emacs 29, requiring Emacs 30 minimum in my package.

Consider the case where one package requires a bleeding-edge version of a built-in package, but installing that new built-in package breaks several other packages which do not rely on the latest version.
The user will have to make a decision no matter the default.

In theory, this might be. But in practice, using Straight "just works" whereas using Elpaca does not. Part of the issue is that Emacs ecosystem does not really have versioning infrastructure -- the name "MELPA Stable" is misleading, for example. So perhaps it is jumping the gun to worry overmuch about retaining the "stable" version of something. As long as the ecosystem is like this, you get the best stability by staying on everyone's master branch anyway.

[progfolio]

If a pacakge literally lists (transient "0.7.4") in its Package-Requires, it seems to me a bug if a package manager does not react to that.

It does react. It will signal an error if the dependency cannot be satisfied. From there the user can decide whether they'd like to update transient or downgrade/skip the package which requires a newer version.

In theory, this might be. But in practice, using Straight "just works" whereas using Elpaca does not.

Straight ignores the problem altogether, which leads to confusing bugs down the road for users and package maintainers.

[meedstrom]

It does react. It will signal an error if the dependency cannot be satisfied.

Oh, I must've forgotten that. Sorry, that's good then.