-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for nixos depexts #5332
base: master
Are you sure you want to change the base?
Conversation
10d7604
to
3434460
Compare
@@ -129,6 +130,7 @@ let family ~env () = | |||
failwith | |||
"External dependency handling for macOS requires either \ | |||
MacPorts or Homebrew - neither could be found" | |||
| "nixos" -> Nixos |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This does lead to the question of what opam will/should do if running on macOS with nix-darwin (as I do). I have both Homebrew and nix-darwin installed -- it would be nice to be able to explicitly configure which depext backend to use rather than have it inferred by opam.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding what it will do, I believe OpamSysPoll.os_family env
will return homebrew
/macports
, so the behavior will be unchanged by this PR.
Regarding what it should do, I don't think there's much to be gained from installing depexts on macOS through nix-env
. My opinion is nix-env
isn't the 'right' way to use Nix, but is a useful function to provide an interface similar to traditional package managers. As such, this PR is useful in that it provides the path of least resistance to getting a project building on a NixOS system with existing OCaml tooling, as opposed to using one of the various projects for building OCaml projects with Nix.
Regarding explicitly configuring which backend to use, opam is already using homebrew by default and macports if homebrew is unavailable. Using nix-env
could be useful in providing a third option (with a very minor change), i.e.:
else if OpamSystem.resolve_command "nix-env" <> None then Some "nix"
Perhaps explicitly configuring which depext backend to use could be another PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Ryan, that all sounds very sensible to me.
Hi Stephen:-) It looks like this addresses #5124. While I had an idea to generate The only thing is I prefer not to use |
@@ -734,6 +839,7 @@ let update ?(env=OpamVariable.Map.empty) () = | |||
| Gentoo -> Some (`AsAdmin "emerge", ["--sync"]) | |||
| Homebrew -> Some (`AsUser "brew", ["update"]) | |||
| Macports -> Some (`AsAdmin "port", ["sync"]) | |||
| Nixos -> Some (`AsUser "nix-channel", ["--update"]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't do anything if the NixOS system is (like mine) using flakes to provide a instance Nixpkgs instances instead of channels. However, we definitely shouldn't call nix flake lock --update-input nixpkgs
from opam so I don't have a better suggestion.
A more general point: does this have opam invoking a command to update the entire system? I notice pacman
is invoked with -Sy
, which is not recommended as refreshing wityhout updating results in partial upgrades. I think this should be -Syu
. Unless there's a good reason not too I'll create a new issue for this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is mostly use for CI systems that really benefit from a non-platform way of updating the system package repository.
A more general point: does this have opam invoking a command to update the entire system? I notice
pacman
is invoked with-Sy
, which is not recommended as refreshing wityhout updating results in partial upgrades. I think this should be-Syu
. Unless there's a good reason not too I'll create a new issue for this.
From opam's point of view this is safe to do because installing a package is done using pacman -Su
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't do anything if the NixOS system is (like mine) using flakes to provide a instance Nixpkgs instances instead of channels.
I feel like I'm always learning new ways to use nix.
Opam doesn't prompt the user before installing packages. I can see how a nix user might be bothered by it invoking |
I think it would, if there's a clean way to do that without effecting other platforms? I think I see some logic for prompting the user in https://github.com/gridbugs/opam/blob/3434460674f4df689dc982971ba9c2e96d349902/src/client/opamSolution.ml#L1146-L1278 but I'm not too familiar with this codebase. |
is slow, and takes the same amount of time as listing all the | ||
packages (~30s). We need to check the availability of multiple | ||
packages, so it's faster to list all packages once and then search | ||
the output for the specified packages. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's worth noting flake hermetic evaluation allows nix expression caching which could speed this up significantly. Using nix flake show nixpkgs --legacy --json
takes a couple of seconds (after an initial long run) but lists packages (with Attr Names and Names [0]) and for all systems. nix eval nixpkgs\#pkgs --apply builtins.attrNames
is also very fast, but doesn't give us PNames [0], which I believe is what nix-env
uses. Trying to be a bit more sophisticated with nix eval nixpkgs\#pkgs --apply "pkgs: pkgs.lib.attrsets.mapAttrsToList (n: v: v.pname ) pkgs"
errors out failing to evaluate derivations.
So I don't have a better suggestion :-) Just adding this for posterity.
[0] For understanding the different names at play: https://discourse.nixos.org/t/clarification-on-package-names-and-versions/9819
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I've just found the --attr/-A
option of nix-env
which allows us to install packages by their attr name, a list of which can be obtained from nix eval nixpkgs\#pkgs --apply builtins.attrNames
. This is also beneficial as the attr name is guaranteed to be globally unique, and some derivations might not have a pname.
An aside: there are 114
instances nixos
depext
dependencies in opam-repository
. I'm wondering if this PR will only pick up these depext
s or if it can pick up non-nixos
depext
s.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another option is just to call nix search nixpkgs\#<attr-name>
for every package, and install them with nix-env -iA <attr-name>
. Calling nix search
for every package is not expensive due to nix expression caching.
Having said all of this, the nix
command interfaces are unstable, and flakes are still experimental, and adding two nix backends (for flakes and non-flakes) adds additional complexity, so perhaps it's best to keep using nix-env
.
I'm thinking about the correct way to prompt the user before installing packages, or if this is the right way to solve this problem at all. The original reason I made this PR is because I was following this mirage tutorial and opam crashed while attempting to install depexts because I'm running nixos and opam doesn't know how to interact with the package manager. The specific command that caused the crash was Worth noting is that I've also raised tarides/opam-monorepo#344, which makes I think our choices are:
The third option appeals to me because it would let me continue managing packages declaratively, but opam would still be telling me what packages it expects to be available. But it appears that |
If you use
As @Leonidas-from-XIV says in tarides/opam-monorepo#344 I think we want to minimize reliance on global mutable state. The
Short of just trying to invoke a command (which through conversations with David Allsop I think this is how opam currently checks installation status?) I don't know a good way to do this. However, a slight twist on the third option: we could also generate a |
Speaking of mirage, I'd also encourage you to try hillingar. It invokes the opam solver, so should 'just work'. An example is here mirage-hello-hillingar. The only thing is it doesn't pick up depexts in the monorepo yet so they need to be manually added to the flake. |
Also, @sternenseemann makes a good point here that while binary depexts may work with |
Hej everyone! Finally catching up this PR. Thanks all for these highly needed inputs (there is no nix user in opam team). Here's some answer / inputs on the discussion.
After discussing it in dev meeting, we end up with the conclusion that we should keep that PR for
Opam always prompt before calling system package manager, we don't want to install something in user's computer without validation. Even with
Speed is crucial here, for several commands this installed/availability computation is done. it can't take more then 1-2s at most.
Better not rely on an experimental feature as no everyone is on opam master branch (nor last release) ; and it's better to have more coverage on packages detection. Which one should be used then? |
While this is true, it is to be noted that we do not show the command that opam is going to run. I think we'd want that here (and that would be an improvement in my opinion) |
"failed with unexpected exit code %d running command:\n %s\n\nOutput:\n%s" | ||
code (String.concat " " (nix_env::args)) (String.concat "\n" output) | ||
in | ||
let all_pnames_of_nix_package_json : OpamJson.t -> string list option = function |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't be faster to directly parse output (list filtering / regexp) than work on parsed json output ? see Arch
for example.
let not_installed = OpamSysPkg.Set.filter (Fun.negate is_package_installed) packages in | ||
if OpamSysPkg.Set.is_empty not_installed then | ||
OpamSysPkg.Set.empty, OpamSysPkg.Set.empty | ||
else | ||
let not_installed_by_str = | ||
OpamSysPkg.Set.elements not_installed | ||
|> List.map (fun pkg -> OpamSysPkg.to_string pkg, pkg) | ||
|> OpamStd.String.Map.of_list | ||
in | ||
let available_to_install = | ||
get_all_available_package_names () | ||
|> List.filter_map (fun pname -> OpamStd.String.Map.find_opt pname not_installed_by_str) | ||
|> OpamSysPkg.Set.of_list | ||
in | ||
(available_to_install, OpamSysPkg.Set.diff not_installed available_to_install) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Internal function compute_sets simplifies this computation.
the output for the specified packages. | ||
*) | ||
let args = ["--query"; "--available"; "--json"] in | ||
let (code, output_lines) = run_command nix_env args in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason to not use run_query_command
?
@@ -678,6 +782,7 @@ let install_packages_commands_t ?(env=OpamVariable.Map.empty) sys_packages = | |||
[`AsAdmin "port", yes ["-N"] ("install"::packages)], | |||
None | |||
| Netbsd -> [`AsAdmin "pkgin", yes ["-y"] ("install" :: packages)], None | |||
| Nixos -> [`AsUser "nix-env", "--install" :: packages], None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does nix-env
have a non-interactive option?
True, but the issue here is to acknowledge user that |
Makes sense! I do think this nix shell solution (as opposed to the
That's great to hear.
This makes flake nix expression caching all the more attractive...
I think we should take an opinionated stance and use flakes given the performance benefits (through expression caching). While they are technically still experimental, they are widely adopted in the Nix ecosystem. We can avoid using the unstable command line APIs if we aren't using the |
To clarify, what I think @gridbugs is pondering, and what I'm proposing, is creating a file that describes the depexts to be used. See here: https://nixos.wiki/wiki/Development_environment_with_nix-shell. There's many idiomatic arguments for this but the key reason I see for doing it is that library depexts installed with The only difficulty with this is it requires quite a different model of interacting with the system than traditional package managers, and I don't know much refactoring would be required to implement it. EDIT: This might be similar to https://github.com/DeterminateSystems/riff |
Also, looking at this from a birds eye view, what we're trying to do here is to get existing OCaml tooling working on a NixOS system. However, as we're not building inside a Nix derivation, we don't get the benefits of reproducible builds (in the system state sense - not strict binary compatibility sense). It may be worth checking out one of the number of projects for building OCaml/Opam projects on Nix, notably tweag/opam-nix (which uses the opam solver and should 'just work'), or using nix-ocaml/nix-overlays (if you want to package your own dependencies in Nix). |
Any progress on this? |
To expand on this, a real world example I always have issues with is the GNU Multiple Precision Arithmetic Library ( A workaround is to use: Whereas using e.g.
This is related to NixOS/nix#8386. |
No description provided.