Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for nixos depexts #5332

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gridbugs
Copy link
Contributor

No description provided.

@@ -129,6 +130,7 @@ let family ~env () =
failwith
"External dependency handling for macOS requires either \
MacPorts or Homebrew - neither could be found"
| "nixos" -> Nixos
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does lead to the question of what opam will/should do if running on macOS with nix-darwin (as I do). I have both Homebrew and nix-darwin installed -- it would be nice to be able to explicitly configure which depext backend to use rather than have it inferred by opam.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding what it will do, I believe OpamSysPoll.os_family env will return homebrew/macports, so the behavior will be unchanged by this PR.

Regarding what it should do, I don't think there's much to be gained from installing depexts on macOS through nix-env. My opinion is nix-env isn't the 'right' way to use Nix, but is a useful function to provide an interface similar to traditional package managers. As such, this PR is useful in that it provides the path of least resistance to getting a project building on a NixOS system with existing OCaml tooling, as opposed to using one of the various projects for building OCaml projects with Nix.

Regarding explicitly configuring which backend to use, opam is already using homebrew by default and macports if homebrew is unavailable. Using nix-env could be useful in providing a third option (with a very minor change), i.e.:

else if OpamSystem.resolve_command "nix-env" <> None then Some "nix"

Perhaps explicitly configuring which depext backend to use could be another PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Ryan, that all sounds very sensible to me.

@RyanGibb
Copy link
Contributor

Hi Stephen:-)

It looks like this addresses #5124.

While I had an idea to generate shell.nix files from opam, I think using nix-env is more aligned with opam's existing approach, so this seems like a great addition to me.

The only thing is I prefer not to use nix-env, instead configuring per-project environments or installing packages globally declaratively. Does opam prompt the user before installing packages by default?

@@ -734,6 +839,7 @@ let update ?(env=OpamVariable.Map.empty) () =
| Gentoo -> Some (`AsAdmin "emerge", ["--sync"])
| Homebrew -> Some (`AsUser "brew", ["update"])
| Macports -> Some (`AsAdmin "port", ["sync"])
| Nixos -> Some (`AsUser "nix-channel", ["--update"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't do anything if the NixOS system is (like mine) using flakes to provide a instance Nixpkgs instances instead of channels. However, we definitely shouldn't call nix flake lock --update-input nixpkgs from opam so I don't have a better suggestion.

A more general point: does this have opam invoking a command to update the entire system? I notice pacman is invoked with -Sy, which is not recommended as refreshing wityhout updating results in partial upgrades. I think this should be -Syu. Unless there's a good reason not too I'll create a new issue for this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly use for CI systems that really benefit from a non-platform way of updating the system package repository.

A more general point: does this have opam invoking a command to update the entire system? I notice pacman is invoked with -Sy, which is not recommended as refreshing wityhout updating results in partial upgrades. I think this should be -Syu. Unless there's a good reason not too I'll create a new issue for this.

From opam's point of view this is safe to do because installing a package is done using pacman -Su.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't do anything if the NixOS system is (like mine) using flakes to provide a instance Nixpkgs instances instead of channels.

I feel like I'm always learning new ways to use nix.

@gridbugs
Copy link
Contributor Author

The only thing is I prefer not to use nix-env, instead configuring per-project environments or installing packages globally declaratively. Does opam prompt the user before installing packages by default?

Opam doesn't prompt the user before installing packages. I can see how a nix user might be bothered by it invoking nix-env by surprise, more so than users of other operating systems. Would it make sense to add a prompt before installing any packages on nixos?

@rjbou rjbou self-requested a review October 31, 2022 14:02
@RyanGibb
Copy link
Contributor

The only thing is I prefer not to use nix-env, instead configuring per-project environments or installing packages globally declaratively. Does opam prompt the user before installing packages by default?

Opam doesn't prompt the user before installing packages. I can see how a nix user might be bothered by it invoking nix-env by surprise, more so than users of other operating systems. Would it make sense to add a prompt before installing any packages on nixos?

I think it would, if there's a clean way to do that without effecting other platforms?

I think I see some logic for prompting the user in https://github.com/gridbugs/opam/blob/3434460674f4df689dc982971ba9c2e96d349902/src/client/opamSolution.ml#L1146-L1278 but I'm not too familiar with this codebase.

is slow, and takes the same amount of time as listing all the
packages (~30s). We need to check the availability of multiple
packages, so it's faster to list all packages once and then search
the output for the specified packages.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth noting flake hermetic evaluation allows nix expression caching which could speed this up significantly. Using nix flake show nixpkgs --legacy --json takes a couple of seconds (after an initial long run) but lists packages (with Attr Names and Names [0]) and for all systems. nix eval nixpkgs\#pkgs --apply builtins.attrNames is also very fast, but doesn't give us PNames [0], which I believe is what nix-env uses. Trying to be a bit more sophisticated with nix eval nixpkgs\#pkgs --apply "pkgs: pkgs.lib.attrsets.mapAttrsToList (n: v: v.pname ) pkgs" errors out failing to evaluate derivations.

So I don't have a better suggestion :-) Just adding this for posterity.

[0] For understanding the different names at play: https://discourse.nixos.org/t/clarification-on-package-names-and-versions/9819

Copy link
Contributor

@RyanGibb RyanGibb Oct 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I've just found the --attr/-A option of nix-env which allows us to install packages by their attr name, a list of which can be obtained from nix eval nixpkgs\#pkgs --apply builtins.attrNames. This is also beneficial as the attr name is guaranteed to be globally unique, and some derivations might not have a pname.

An aside: there are 114 instances nixos depext dependencies in opam-repository. I'm wondering if this PR will only pick up these depexts or if it can pick up non-nixos depexts.

Copy link
Contributor

@RyanGibb RyanGibb Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option is just to call nix search nixpkgs\#<attr-name> for every package, and install them with nix-env -iA <attr-name>. Calling nix search for every package is not expensive due to nix expression caching.

Having said all of this, the nix command interfaces are unstable, and flakes are still experimental, and adding two nix backends (for flakes and non-flakes) adds additional complexity, so perhaps it's best to keep using nix-env.

@gridbugs
Copy link
Contributor Author

gridbugs commented Nov 2, 2022

The only thing is I prefer not to use nix-env, instead configuring per-project environments or installing packages globally declaratively. Does opam prompt the user before installing packages by default?

Opam doesn't prompt the user before installing packages. I can see how a nix user might be bothered by it invoking nix-env by surprise, more so than users of other operating systems. Would it make sense to add a prompt before installing any packages on nixos?

I think it would, if there's a clean way to do that without effecting other platforms?

I think I see some logic for prompting the user in https://github.com/gridbugs/opam/blob/3434460674f4df689dc982971ba9c2e96d349902/src/client/opamSolution.ml#L1146-L1278 but I'm not too familiar with this codebase.

I'm thinking about the correct way to prompt the user before installing packages, or if this is the right way to solve this problem at all. The original reason I made this PR is because I was following this mirage tutorial and opam crashed while attempting to install depexts because I'm running nixos and opam doesn't know how to interact with the package manager.

The specific command that caused the crash was opam monorepo depext -y ... in the depext-lockfile target of the makefile generated by mirage. My initial reaction was to allow opam to handle nixos depexts (by invoking nix-env --install ...) (ie. this PR). The problem with this is installing packages in a non-declarative way (ie. with nix-env) is frowned upon by some nix users. We could maybe get around this by adding an interactive prompt to confirm the installation (on nixos only), but I think we would want to allow passing -y to override this prompt so the command can be used in non-interactive environments, and notably the original command that caused me grief was passing -y.

Worth noting is that I've also raised tarides/opam-monorepo#344, which makes opam monorepo depexts respect the depext setting in the global opam config which will solve my immediate problem.

I think our choices are:

  • keep the behaviour prior to this PR and rely on nixos users to disallow depexts via the global opam config
  • have opam invoke nix-env to install packages, possibly with a prompt to confirm, but which is overriden by passing -y
  • have opam check what packages need to be installed, and then exit with a message instructing the user to install those packages by one of the many methods available to nix users

The third option appeals to me because it would let me continue managing packages declaratively, but opam would still be telling me what packages it expects to be available. But it appears that nix-env --query doesn't know about packages that were installed by other means (e.g. if I do nix-shell -p cmake and then nix-env --query cmake it doesn't get picked up). @RyanGibb is there a way to check whether a package is available in the current shell?

@RyanGibb
Copy link
Contributor

RyanGibb commented Nov 2, 2022

The specific command that caused the crash was opam monorepo depext -y ... in the depext-lockfile target of the makefile generated by mirage.

If you use mirage configure -t <target> --no-depext it will omit the depext logic from the makefile.

I think our choices are:

* keep the behaviour prior to this PR and rely on nixos users to disallow depexts via the global opam config

As @Leonidas-from-XIV says in tarides/opam-monorepo#344 I think we want to minimize reliance on global mutable state. The mirage --no-depext option might be sufficient for your immediate problem?

* have opam invoke `nix-env` to install packages, possibly with a prompt to confirm, but which is overriden by passing `-y`
* have opam check what packages need to be installed, and then exit with a message instructing the user to install those packages by one of the many methods available to nix users

The third option appeals to me because it would let me continue managing packages declaratively, but opam would still be telling me what packages it expects to be available. But it appears that nix-env --query doesn't know about packages that were installed by other means (e.g. if I do nix-shell -p cmake and then nix-env --query cmake it doesn't get picked up). @RyanGibb is there a way to check whether a package is available in the current shell?

Short of just trying to invoke a command (which through conversations with David Allsop I think this is how opam currently checks installation status?) I don't know a good way to do this.

However, a slight twist on the third option: we could also generate a shell.nix so all the user has to do is call nix-shell (or the flake equivalents flake.nix with a devShell output and call nix develop). I.e., #5124. This would remove the need to probe package availability. We would just add all depexts to this file, if they're already installed they won't be installed again.

@RyanGibb
Copy link
Contributor

RyanGibb commented Nov 2, 2022

Speaking of mirage, I'd also encourage you to try hillingar. It invokes the opam solver, so should 'just work'. An example is here mirage-hello-hillingar.

The only thing is it doesn't pick up depexts in the monorepo yet so they need to be manually added to the flake.

@RyanGibb
Copy link
Contributor

RyanGibb commented Nov 2, 2022

Also, @sternenseemann makes a good point here that while binary depexts may work with nix-env, libraries won't be found by the linker. I think nix-shell/nix develop will set appropriate flags to remedy this.

@rjbou
Copy link
Collaborator

rjbou commented Nov 2, 2022

Hej everyone! Finally catching up this PR. Thanks all for these highly needed inputs (there is no nix user in opam team). Here's some answer / inputs on the discussion.

This does lead to the question of what opam will/should do if running on macOS with nix-darwin (as I do). I have both Homebrew and nix-darwin installed -- it would be nice to be able to explicitly configure which depext backend to use rather than have it inferred by opam.

After discussing it in dev meeting, we end up with the conclusion that we should keep that PR for nixos distribution only. Nix can be installed and used in several platform, and to do it the good way, we have to implement first os-package-repository variable (see #4440) to be able to cleanly and clearly specify the wanted package manager to use and its repository.

Opam doesn't prompt the user before installing packages

Opam always prompt before calling system package manager, we don't want to install something in user's computer without validation. Even with --yes the prompt if shown, for automatisation (CI scripts), there is option --confirm-level=unsafe-yes that runs in a non interactive mode (opam and underlying package manager). You can see an example of prompt in this test (for test, system package manager command is echo)

The command for querying whether a single package is available is slow, and takes the same amount of time as listing all the packages (~30s)

Speed is crucial here, for several commands this installed/availability computation is done. it can't take more then 1-2s at most.

Having said all of this, the nix command interfaces are unstable, and flakes are still experimental, and adding two nix backends (for flakes and non-flakes) adds additional complexity, so perhaps it's best to keep using nix-env.

binary depexts may work with nix-env, libraries won't be found by the linker. I think nix-shell/nix develop will set appropriate flags to remedy this.

Better not rely on an experimental feature as no everyone is on opam master branch (nor last release) ; and it's better to have more coverage on packages detection. Which one should be used then?

@kit-ty-kate
Copy link
Member

Opam always prompt before calling system package manager, we don't want to install something in user's computer without validation. Even with --yes the prompt if shown, for automatisation (CI scripts), there is option --confirm-level=unsafe-yes that runs in a non interactive mode (opam and underlying package manager). You can see an example of prompt in this test (for test, system package manager command is echo)

While this is true, it is to be noted that we do not show the command that opam is going to run. I think we'd want that here (and that would be an improvement in my opinion)

"failed with unexpected exit code %d running command:\n %s\n\nOutput:\n%s"
code (String.concat " " (nix_env::args)) (String.concat "\n" output)
in
let all_pnames_of_nix_package_json : OpamJson.t -> string list option = function
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be faster to directly parse output (list filtering / regexp) than work on parsed json output ? see Arch for example.

Comment on lines +688 to +702
let not_installed = OpamSysPkg.Set.filter (Fun.negate is_package_installed) packages in
if OpamSysPkg.Set.is_empty not_installed then
OpamSysPkg.Set.empty, OpamSysPkg.Set.empty
else
let not_installed_by_str =
OpamSysPkg.Set.elements not_installed
|> List.map (fun pkg -> OpamSysPkg.to_string pkg, pkg)
|> OpamStd.String.Map.of_list
in
let available_to_install =
get_all_available_package_names ()
|> List.filter_map (fun pname -> OpamStd.String.Map.find_opt pname not_installed_by_str)
|> OpamSysPkg.Set.of_list
in
(available_to_install, OpamSysPkg.Set.diff not_installed available_to_install)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Internal function compute_sets simplifies this computation.

the output for the specified packages.
*)
let args = ["--query"; "--available"; "--json"] in
let (code, output_lines) = run_command nix_env args in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to not use run_query_command?

@@ -678,6 +782,7 @@ let install_packages_commands_t ?(env=OpamVariable.Map.empty) sys_packages =
[`AsAdmin "port", yes ["-N"] ("install"::packages)],
None
| Netbsd -> [`AsAdmin "pkgin", yes ["-y"] ("install" :: packages)], None
| Nixos -> [`AsUser "nix-env", "--install" :: packages], None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does nix-env have a non-interactive option?

@rjbou
Copy link
Collaborator

rjbou commented Nov 2, 2022

Opam always prompt before calling system package manager, we don't want to install something in user's computer without validation. Even with --yes the prompt if shown, for automatisation (CI scripts), there is option --confirm-level=unsafe-yes that runs in a non interactive mode (opam and underlying package manager). You can see an example of prompt in this test (for test, system package manager command is echo)

While this is true, it is to be noted that we do not show the command that opam is going to run. I think we'd want that here (and that would be an improvement in my opinion)

True, but the issue here is to acknowledge user that nix-env will be used (cf. #5332 (comment)). Packages are listed just above the menu. If you want to see the command before running it, it's not intuitive, but there is a way, see #5053 (comment) (we can discuss depext menu UI in another issue).

@RyanGibb
Copy link
Contributor

RyanGibb commented Nov 2, 2022

After discussing it in dev meeting, we end up with the conclusion that we should keep that PR for nixos distribution only. Nix can be installed and used in several platform, and to do it the good way, we have to implement first os-package-repository variable (see #4440) to be able to cleanly and clearly specify the wanted package manager to use and its repository.

Makes sense! I do think this nix shell solution (as opposed to the nix-env solution) could benefit other platforms (e.g. MacOS) by providing a more reproducible way to provide depexts, but it sounds like this is blocked by some opam limitations.

Opam always prompt before calling system package manager, we don't want to install something in user's computer without validation. Even with --yes the prompt if shown, for automatisation (CI scripts), there is option --confirm-level=unsafe-yes that runs in a non interactive mode (opam and underlying package manager). You can see an example of prompt in this test (for test, system package manager command is echo)

That's great to hear.

The command for querying whether a single package is available is slow, and takes the same amount of time as listing all the packages (~30s)

Speed is crucial here, for several commands this installed/availability computation is done. it can't take more then 1-2s at most.

This makes flake nix expression caching all the more attractive...

Having said all of this, the nix command interfaces are unstable, and flakes are still experimental, and adding two nix backends (for flakes and non-flakes) adds additional complexity, so perhaps it's best to keep using nix-env.

binary depexts may work with nix-env, libraries won't be found by the linker. I think nix-shell/nix develop will set appropriate flags to remedy this.

Better not rely on an experimental feature as no everyone is on opam master branch (nor last release) ; and it's better to have more coverage on packages detection. Which one should be used then?

I think we should take an opinionated stance and use flakes given the performance benefits (through expression caching). While they are technically still experimental, they are widely adopted in the Nix ecosystem. We can avoid using the unstable command line APIs if we aren't using the nix-env solution. I would be interested to hear any arguments against this, though.

@RyanGibb
Copy link
Contributor

RyanGibb commented Nov 2, 2022

Opam always prompt before calling system package manager, we don't want to install something in user's computer without validation. Even with --yes the prompt if shown, for automatisation (CI scripts), there is option --confirm-level=unsafe-yes that runs in a non interactive mode (opam and underlying package manager). You can see an example of prompt in this test (for test, system package manager command is echo)

While this is true, it is to be noted that we do not show the command that opam is going to run. I think we'd want that here (and that would be an improvement in my opinion)

True, but the issue here is to acknowledge user that nix-env will be used (cf. #5332 (comment)). Packages are listed just above the menu. If you want to see the command before running it, it's not intuitive, but there is a way, see #5053 (comment) (we can discuss depext menu UI in another issue).

To clarify, what I think @gridbugs is pondering, and what I'm proposing, is creating a file that describes the depexts to be used. See here: https://nixos.wiki/wiki/Development_environment_with_nix-shell. There's many idiomatic arguments for this but the key reason I see for doing it is that library depexts installed with nix-env won't be found by the linker.

The only difficulty with this is it requires quite a different model of interacting with the system than traditional package managers, and I don't know much refactoring would be required to implement it.

EDIT: This might be similar to https://github.com/DeterminateSystems/riff

@RyanGibb
Copy link
Contributor

RyanGibb commented Nov 2, 2022

Also, looking at this from a birds eye view, what we're trying to do here is to get existing OCaml tooling working on a NixOS system. However, as we're not building inside a Nix derivation, we don't get the benefits of reproducible builds (in the system state sense - not strict binary compatibility sense). It may be worth checking out one of the number of projects for building OCaml/Opam projects on Nix, notably tweag/opam-nix (which uses the opam solver and should 'just work'), or using nix-ocaml/nix-overlays (if you want to package your own dependencies in Nix).

@Profpatsch
Copy link

Any progress on this?

@RyanGibb
Copy link
Contributor

RyanGibb commented Sep 7, 2023

There's many idiomatic arguments for this but the key reason I see for doing it is that library depexts installed with nix-env won't be found by the linker.

To expand on this, a real world example I always have issues with is the GNU Multiple Precision Arithmetic Library (gmp). Installing it any mechanism (nix-env or flakes) doesn't set NIX_CFLAGS_COMPILE, which is only done in a build environment (in a derivation, or a Nix shell).

A workaround is to use:
CFLAGS="-isystem ``nix eval --inputs-from /etc/nixos --raw nixpkgs#gmpxx.dev``/include" opam install conf-gmp

Whereas using e.g. nix develop fails:

$ nix develop nixpkgs#gmpxx.dev
$ opam install conf-gmp
...
#=== ERROR while compiling conf-gmp.4 =========================================#
# context     2.1.4 | linux/x86_64 |  | https://opam.ocaml.org#1133515a
# path        ~/projects/hoke/_opam/.opam-switch/build/conf-gmp.4
# command     /nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/sh -exc cc -c $CFLAGS -I/usr/local/include test.c
# exit-code   1
# env-file    ~/.opam/log/conf-gmp-1515984-5ce62d.env
# output-file ~/.opam/log/conf-gmp-1515984-5ce62d.out
### output ###
# + cc -c -I/usr/local/include test.c
# test.c:1:10: fatal error: gmp.h: No such file or directory
#     1 | #include <gmp.h>
#       |          ^~~~~~~
# compilation terminated.

This is related to NixOS/nix#8386.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants