From 641b83172cac4a68693a6852fe2da9f317bae9b0 Mon Sep 17 00:00:00 2001 From: Sukant Hajra Date: Tue, 21 May 2024 10:50:08 -0500 Subject: [PATCH] WIP: workspace prep for jj --- .github/workflows/ci.yml | 6 +- doc/internal/nix-introduction-compat.org | 46 +-- doc/internal/nix-usage-flakes-optional.org | 10 +- doc/internal/params.el | 4 +- doc/nix-installation.org | 50 +-- doc/nix-introduction.org | 389 +++++++++++---------- doc/nix-language.org | 188 +++++----- doc/nix-usage-flakes.org | 270 +++++++------- doc/nix-usage-noflakes.org | 199 ++++++----- 9 files changed, 608 insertions(+), 554 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9f38a37..c917831 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,9 +10,9 @@ jobs: name: ${{ matrix.os }} build runs-on: ${{ matrix.os }} steps: - - uses: actions/checkout@v2 - - uses: cachix/install-nix-action@v17 - - uses: cachix/cachix-action@v10 + - uses: actions/checkout@v4 + - uses: cachix/install-nix-action@v27 + - uses: cachix/cachix-action@v15 with: name: shajra signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}' diff --git a/doc/internal/nix-introduction-compat.org b/doc/internal/nix-introduction-compat.org index a73fad4..e4228ed 100644 --- a/doc/internal/nix-introduction-compat.org +++ b/doc/internal/nix-introduction-compat.org @@ -6,19 +6,19 @@ project-dependent. * Helping non-flakes users -A few users make work in organizations or contribute to projects that disallow +A few users work in organizations or contribute to projects that disallow experimental features such as flakes. -To buffer this compromise, this project uses and encourages the use of the -[[nix-flake-compat][flake-compat]] project, which enables an end user who has opted not to enable -flakes to at least access the flake's contents, packages or otherwise. +For these users, this project uses and encourages the use of the [[nix-flake-compat][flake-compat]] +project, which enables an end user who has opted not to enable flakes to at +least access the flake's contents, packages or otherwise. -With flake-compat, end users will have a normal (non-flake) Nix expression they +With flake-compat, end users will have a regular (non-flake) Nix expression they can evaluate. However, since dependencies are managed with flakes, the project maintainer must have flakes enabled to manage dependencies (for example, updating to the latest dependencies with =nix flake update=). -* Documenting an end user experience +* Documenting an end-user experience To deal with the transition of the Nix community to flake, this project provides two user guides: @@ -26,38 +26,40 @@ two user guides: - [[file:nix-usage-flakes.org][Nix Usage with Flakes (Recommended) ]] - [[file:nix-usage-noflakes.org][Nix Usage without Flakes]] -Links generally steer users to the recommended guide, which then links users to -the non-flakes guide if they have the interest or need. +Links generally steer users to the recommended flakes guide, which then links +users to the non-flakes guide if they have the interest or need. The non-flakes guide intentionally avoids commands like =nix-shell= and -=nix-channel=. These commands lead users to setting the =NIX_PATH= environment -variable, which can lead to unreliable builds. +=nix-channel=. These commands lead users to set the =NIX_PATH= environment +variable, which can lead to unreliable builds. These are the pitfalls that +motivated the design of flakes. -Though this guide avoid the =flakes= experimental feature, it still invites end -users to use the experimental =nix-command= to get the following subcommands: +Though this non-flakes guide avoids the =flakes= experimental feature, it still +invites end users to use the experimental =nix-command= to get the following +subcommands: - =nix search= - =nix shell= - =nix run= -In general, the non-flakes guide only explains usage of experimental =nix= -subcommands when there exist no other alternatives, or when the alternatives are +In general, the non-flakes guide only explains the usage of experimental =nix= +subcommands when there exist no other alternatives or when the alternatives are considered worse for new users. -=nix search= simply has no good alternative within the set of non-experimental -Nix tools, but it's too useful to not tell users about. Again, this is an +For example, =nix search= has no alternative within the set of non-experimental +Nix tools, and it's too helpful not to tell users about it. Again, this is an example of the Nix community leading users to experimental features. -=nix shell= and =nix run= are shown as improved alternatives to =nix-shell=. -=nix-shell= is a complicated tool that has been historically used for a lot of -different purposes: +Additionally, =nix shell= and =nix run= are shown as improved alternatives to +=nix-shell=. =nix-shell= is a complicated tool that has been historically used +for a lot of different purposes: - debugging the build environments of packages - creating a developer environment for a package (=nix develop= does this - better, but for only for flakes) + better, but only for flakes) - entering a shell with Nix-build executables on the path (=nix shell= does this better) - running arbitrary commands with Nix-build executables on the path (=nix run= does this better) To cover all of these scenarios, =nix-shell= became so complex it is hard to -explain to new users. =nix-shell= is really only best for debugging builds, -which is beyond the scope of the documentation provided by this project. +explain to new users. =nix-shell= is only best for debugging builds, which is +beyond the scope of the documentation provided by this project. diff --git a/doc/internal/nix-usage-flakes-optional.org b/doc/internal/nix-usage-flakes-optional.org index 39a4f65..1629de2 100644 --- a/doc/internal/nix-usage-flakes-optional.org +++ b/doc/internal/nix-usage-flakes-optional.org @@ -5,12 +5,12 @@ project-dependent. #+end_comment This project supports a still-experimental feature of Nix called /flakes/, which -this guide shows users how to use. [[file:../nix-usage-noflakes.org][Another guide]] explains how to do -everything illustrated in this document, but without flakes. +this guide shows users how to use. [[file:nix-usage-noflakes.org][Another guide]] explains how to do everything +illustrated in this document, but without flakes. #+begin_quote -*_NOTE:_* If you're new to flakes, please read the provided [[file:../nix-introduction.org][supplemental +*_NOTE:_* If you're new to flakes, please read the provided [[file:nix-introduction.org][supplemental introduction to Nix]] to understand the experimental nature of flakes and how it -may or may not affect you. Hopefully you'll find these trade-offs acceptable so -you can take advantage of the improved experience flakes offer. +may or may not affect you. Hopefully, you'll find these trade-offs acceptable to +take advantage of the improved experience flakes offer. #+end_quote diff --git a/doc/internal/params.el b/doc/internal/params.el index 65d3f9e..6e3b7df 100644 --- a/doc/internal/params.el +++ b/doc/internal/params.el @@ -10,8 +10,8 @@ ;; run-type must be "executable" (run-target-short . "direnv") (run-target-long . "bin/direnv") - (nix-latest . "2.14") - (nixos-latest . "22.11") + (nix-latest . "2.21") + (nixos-latest . "23.11") (platforms . "\n\ - Linux on x86-64 machines\n\ - MacOS on x86-64 machines\n\ diff --git a/doc/nix-installation.org b/doc/nix-installation.org index 5ff52d2..3c93132 100644 --- a/doc/nix-installation.org +++ b/doc/nix-installation.org @@ -3,7 +3,7 @@ * Org-mode setup :noexport: -This document is written in a project-agnostic way so it can be copied to other +This document is written in a project-agnostic way to be copied to other projects that use Nix. ** Variables @@ -42,23 +42,23 @@ installed in Windows via the Windows Subsystem for Linux (WSL). Installation on WSL may involve steps not covered in this documentation, though. Note, some users may be using [[nixos][NixOS]], a Linux operating system built on top of -Nix. Those users already have Nix and don't need to install it separately. To -use this project, you don't need to use NixOS as well. +Nix. Those users already have Nix and don't need to install it separately. You +don't need to use NixOS to use this project. * Level of commitment/risk Unless you're on NixOS, you're likely already using another package manager for -your operating system already (APT, DNF, etc.). You don't have to worry about -Nix or packages installed by Nix conflicting with anything already on your -system. Running Nix along side other package managers is safe. +your operating system (APT, DNF, etc.). You don't have to worry about Nix or +packages installed by Nix conflicting with anything already on your system. +Running Nix alongside other package managers is safe. -All the files of a Nix package are located under =/nix= a directory, well -isolated from any other package manager. Nix won't touch critical directories -under =/usr= or =/var=. Nix then symlinks files under =/nix= to your home -directory under dot-files like =~/.nix-profile=. There is also some light -configuration under =/etc/nix=. +All the files of a Nix package are located under =/nix= a directory, isolated +from any other package manager. Nix won't touch critical directories under +=/usr= or =/var=. Nix then symlinks files under =/nix= to your home directory +under dot-files like =~/.nix-profile=. There is also some light configuration +under =/etc/nix=. -Hopefully this alleviates any worry about installing a complex program on your +Hopefully, this alleviates any worry about installing a complex program on your machine. Uninstallation is not too much more than deleting everything under =/nix=. @@ -93,9 +93,9 @@ the Nix manual]]. This project pushes built Nix packages to [[cachix][Cachix]] as part of its [[github-actions][continuous integration]]. It's recommended to configure Nix to use shajra.cachix.org as a -Nix /substituter/. Once configured, Nix will pull down pre-built packages from -Cachix, instead of building them locally (potentially saving a lot of time). -This augments Nix's default substituter that pulls from cache.nixos.org. +Nix /substituter/. Once configured, Nix can pull down pre-built packages from +Cachix, instead of building them locally (potentially saving time). Cachix will +augment Nix's default substituter that pulls from cache.nixos.org. You can configure shajra.cachix.org as a supplemental substituter with the following command: @@ -115,7 +115,7 @@ If you've just run a multi-user Nix installation and are not yet a trusted user in =/etc/nix/nix.conf=, this command may not work. But it will report back some options to proceed. -One option sets you up as a trusted user, and installs Cachix configuration for +One option sets you up as a trusted user and installs Cachix configuration for Nix locally at =~/.config/nix/nix.conf=. This configuration will be available immediately, and any subsequent invocation of Nix commands will take advantage of the Cachix cache. @@ -139,18 +139,18 @@ As you can guess, the =flakes= feature enables flakes functionality in Nix. The command-line tool, some of which allow us to work with flakes. If you don't enable experimental features globally, there is a switch to enable -features local to just a single command-line invocation. For example, too use -flakes-related commands we call ~nix --extra-experimental-features 'nix-command -flakes' …~. For users not enabling these features globally, this can be useful -to set to a shell alias. Here's an example that works in most POSIX-compliant -shells: +features local to just a single command-line invocation. For example, to use +flakes-related commands, we call ~nix --extra-experimental-features 'nix-command +flakes' …~. When not configuring globally, setting an alias for this can be +useful. The following command illustrates setting an alias in most +POSIX-compliant shells: #+begin_src sh :eval no alias nix-flakes = nix --extra-experimental-features 'nix-command flakes' #+end_src -As discussed in the introduction, =nix-command= is actually enabled by default. -You don't need to enable it explicitly (though you could disable it). +As discussed in the introduction, =nix-command= is enabled by default. You +don't need to enable it explicitly (though you could disable it). To use flakes there are two things we need to do: 1. make sure the version of Nix we're on is at least 2.4 @@ -166,10 +166,10 @@ nix --version #+end_src #+RESULTS: nix-version -: nix (Nix) 2.11.1 +: nix (Nix) 2.18.1 The easiest way to turn on experimental features is to create a file -=~/.config/nix/nix.conf= if it doesn't already exist, and in it put the +=~/.config/nix/nix.conf= if it doesn't already exist, and in it, put the following line: #+begin_src text :eval no diff --git a/doc/nix-introduction.org b/doc/nix-introduction.org index 71d193e..11fbd3c 100644 --- a/doc/nix-introduction.org +++ b/doc/nix-introduction.org @@ -4,29 +4,30 @@ * About this document This document introduces the [[nix][Nix package manager]] and highlights some motivations -to use Nix. It also covers tradeoffs not only of using Nix, but experimental -features in Nix such as one called /flakes/. +to use Nix. It also covers the tradeoffs of using Nix and experimental features +in Nix, such as /flakes/. This document tries to capture enthusiasm while being honest about frustrations. -Nix is amazing, and a clear pioneer of an architectural approach that users will -come to demand in the future. However, users need clear information up front -where they are likely to face challenges. +Nix is a pioneer of an architectural approach that users will demand in the +future. However, users need clear information up front where they are likely to +face challenges. * Problems addressed by Nix The following sections cover various problems that Nix's architecture addresses. -** Managed build +** Complete build When dealing with a new software project, wrangling dependencies can be a chore. Modern build systems for specific programming languages often don't manage system dependencies. For example, Python's =pip install= will download and -install needed Python dependencies, but may fail if the system doesn't provide C -shared libraries needed for foreign function calls. Complicating matters, -different operating systems have different names for these system packages and -install them with different commands (=apt=, =dnf=, etc.). This makes automation -difficult. Consequently, many software projects only provide documentation as a -surrogate for automation, which creates even more room for error. +install needed Python dependencies but may fail if the system doesn't provide +shared libraries required for foreign function calls. Adding complexity, +different operating systems have differing names for these system packages and +install them with various commands (=apt=, =dnf=, etc.). This variation makes +automation difficult. Consequently, many software projects only provide +documentation as a surrogate for automation, which creates even more room for +error. ** Reliable build @@ -43,39 +44,37 @@ system where something might be built. Once we've built some software and are ready to deploy it, it's not always obvious how to copy this built software to another system. For example, if the software dynamically links to system libraries, we need to know whether those -libraries are on the system we intend to copy to. +libraries are on our target system. ** Version conflicts -Another complication we face is when an operating system only allows one version -of a system library to be installed at a time. When this happens, we have to -make difficult choices if we need two programs that require different versions -of a system dependency. +Another complication we face is when an operating system only allows one +installed version of a system library at a time. When this happens, we may be +forced to make difficult choices if we need two programs requiring different +system dependency versions. ** Polyglot programming -It's also tedious to synthesize libraries and programs from different language +It can be tedious to synthesize libraries and programs from different language ecosystems to make a new program for a unified user experience. For example, the -world of machine learning programming often requires the mixing C/C++, Python, -and even basic shell scripts. These hybrid applications have a tendency to be -fragile. +world of machine learning often requires the mixing of C/C++, Python, and even +basic shell scripts. These hybrid applications tend to be fragile. -** Complete distributed cache of builds +** Distributed cache of builds Various build systems provide repositories for pre-built packages, which helps -users save time by downloading packages instead of building them. What we really -want is this experience, but unified across all programming language ecosystems -and system dependencies. - -Note, this is what traditional package managers like DNF and APT accomplish. But -there's an ergonomic difficulty to turning all software into standard Linux -packages. To start, there are too many Linux distributions with too many package -managers. Secondly, most of the package managers require adherence to a set of -policies for everything to work well together. For example, many distributions -respect the [[fhs][Filesystem Hierarchy Standard (FHS)]]. Confusion around policies have -led many developers to steer away from package managers and towards -container-based technologies like Docker, despite the overhead and drawbacks of -containers. +users save time by downloading packages instead of building them. We want this +experience unified across all programming language ecosystems and system +dependencies. + +Note this is what traditional package managers like DNF and APT accomplish. +However, there's an ergonomic difficulty in turning all software into standard +Linux packages. Firstly, there are too many Linux distributions with too many +package managers. Secondly, most package managers must adhere to policies for +everything to work well together. For example, many distributions respect the +[[fhs][Filesystem Hierarchy Standard (FHS)]]. Confusion around policies has led many +developers to steer away from package managers and toward container-based +technologies like Docker despite the overhead and drawbacks of containers. * Nix at a high level @@ -86,152 +85,151 @@ package manager installed. No other library or system dependency should be required to be installed or configured. Even if we have a library or system dependency installed, it shouldn't interfere -with any build or installation we want to do. +with any build or installation we want to do. Nix builds and installs in its own +directories. -Our build should get everything we need, all the way down to system-level -dependencies, irrespective of which programming language the dependencies has +Our build should get everything we need, all the way down to the system-level +dependencies, irrespective of which programming language the dependencies have been authored in. If anything has been pre-built, we should download a cached result. Above and beyond the problems discussed above, Nix has a precisely deterministic -build, broadly guaranteeing reproducibility. If the package builds on one -system, it should build on all systems, irrespective of what's installed or not. -Furthermore, multiple systems building the same package independently will often +build, generally guaranteeing reproducibility. If the package builds on one +system, it should build on all systems, regardless of what's installed. +Furthermore, multiple systems independently building the same package will often produce bit-for-bit identical builds. -Nix also is able to conveniently copy the transitive closure of a package all -its dependencies ergonomically from one system to another. +Nix is also able to copy the transitive closure of a package's dependencies +ergonomically from one system to another. In broad strokes, Nix is a technology that falls into two categories: - package manager - build tool. -** Nix the package mangager +** Nix the package manager As a package manager, Nix does what most package managers do. Nix provides a suite of command-line tools to search registries of known packages, as well as install and uninstall them. -Packages can provide both executables and plain files alike. Installation just -entails putting these files into a good location for both the package manager -and the user. Nix has an elegant way of storing everything under =/nix/store=, -discussed more below. +Packages can provide both executables and plain files alike. Installation +entails putting these files into a good location for the package manager and the +user. Nix has an elegant way of storing everything under =/nix/store=, discussed +more below. -Importantly, the Nix package manager doesn't differentiate between system-level -installations and user-level installations. All builds and installations are by -nature hermetic and can't conflict with one another. +Notably, the Nix package manager doesn't differentiate between system- and +user-level installations. All packages end up in =/nix/store=. These packages +are hermetic and can't conflict with one another. To save space, packages often +share common elements via symlinks to other packages in =/nix/store=. As a convenience, Nix has tools to help users put the executables provided by -packages provided on their environment's =PATH=. This way, users don't have to -deal with finding executables to call installed in =/nix/store=. +packages on their environment's =PATH=. This way, users don't have to find +executables installed in =/nix/store=. ** Nix the build system -Nix conjoins the features of a package manager with those of a build tool. If a -package or any of its dependencies (including low-level system dependencies) -aren't found in a /Nix substituter/, they are built locally. Otherwise, the -pre-built package and dependencies cached in the Nix substituter are downloaded -rather than built. All we need to build or download any package is the Nix -package manager and a network connection. +Nix combines the features of a package manager with those of a build tool. If a +package or any of its dependent packages (including low-level system +dependencies) aren't found in a /Nix substituter/, Nix builds them locally. +Otherwise, Nix downloads pre-built packages cached in the substituter. We only +need the Nix package manager and a network connection to build or download any +package. Every Nix package is specified by a /Nix expression/, written in a small programming language also called Nix. This expression specifies everything needed to build the package down to the system-level. These expressions are saved in files with a ".nix" extension. -Some software provides these Nix expressions alongside as part of their source. -If some software doesn't provide a Nix expression, you can always use an +Nix-friendly software will provide these Nix expressions as part of their +source. If some software doesn't offer a Nix expression, you can always use an externally authored expression. -What makes Nix special is that these expressions specify a way to build that's +What makes Nix unique is that these expressions specify a way to build that's - precise - repeatable - guaranteed not to conflict with anything already installed For some, it's easy to miss the degree to which Nix-built packages are precise -and repeatable. If you build a package from a Nix expression on one system, and +and repeatable. If you build a package from a Nix expression on one system and then build the same expression on a system of the same architecture, you should get the same result. In most cases, the built artifacts will be identical bit-for-bit. -This degree of precision is accomplished by a system of thorough hashing. In -Nix, the dependencies needed to build packages are also themselves Nix packages. -Every Nix expression has an associated hash that is calculated from the hashes -of package's dependencies and build instructions. When we change this dependency +A system of thorough hashing accomplishes this degree of precision. In Nix, the +dependencies needed to build packages are also themselves Nix packages. Every +Nix expression has an associated hash calculated from the hashes of the +package's dependencies and build instructions. When we change this dependency (even if only by a single bit), the hash for the Nix expression changes. This -cascades to a different calculated hash for any package relying on this -dependency. But if nothing changes, the same hashes will be calculated on all -systems. +new hash cascades to a different calculated hash for any package relying on this +dependency. But if nothing changes, all systems will calculate identical hashes. -The repeatability and precision of Nix forms the basis of how substituters are +The repeatability and precision of Nix form the basis of how substituters are trusted as caching services across the world. It also allows us to trust remote -builds more easily, without worrying about deviations of environment +builds more easily without worrying about deviations in environment configuration. -Nix has central a substituter at https://cache.nixos.org, but there are +Nix has a central substituter at https://cache.nixos.org, but there are third-party ones as well, like [[cachix][Cachix]]. Before building a package, the hash for the package is calculated. If any configured substituter has a build for the hash, it's pulled down as a substitute. A certificate-based protocol is used to -establish trust of substituters. Between this protocol, and the algorithm for +establish the trust of substituters. Between this protocol and the algorithm for calculating hashes in Nix, you can have confidence that a package pulled from a substituter will be identical to what you would have built locally. Finally, all packages are stored in =/nix/store= by their hash. This simple -scheme allows us to have multiple versions of the same package installed with no +scheme allows us to install multiple versions of the same package without conflicts. References to dependencies all point back to the desired version in -=/nix/store= they need. This is not to say that running multiple programs -concurrently based on different versions can't cause problems, but at least the -flexibility to do so is in the user's hands. +=/nix/store= they need. Though Nix has not eliminated the risk of concurrently +running different versions of the same program, at least the flexibility to do +so is in the user's hands. ** Nixpkgs -Nix expressions help us create extremely controlled environments within which we -can build packages precisely. However, Nix still calls the conventional build -tools of various programming language ecosystems. Under the cover, Nix is -ultimately an precisely controlled execution of Bash scripts orchestrating these -tools. +Nix expressions help us create highly controlled environments to build packages +precisely. However, Nix still calls the conventional build tools of various +programming language ecosystems. Under the cover, Nix is ultimately a strictly +controlled execution of Bash scripts orchestrating these tools. -To keep the Nix expressions for each package concise, the Nix community curates -a [[nixpkgs][Git repository of Nix expressions called Nixpkgs]]. Most Nix expressions for -packages will start with a snapshot of Nixpkgs as a dependency, which provides -library support to help keep Nix expressions compact. +The Nix community curates a [[nixpkgs][Git repository of Nix expressions called Nixpkgs]]. +This repository has Nix expressions for all the packages provided by the [[nixos][NixOS]] +operating system, as well as common Nix expressions used to build packages. -This way, the complexity of shell scripting and calls to language-specific -tooling can be kept largely hidden away from general Nix authors. The Nix -language lets us work with packages from any language ecosystem in a uniform -way. +Most Nix expressions for packages will start with a snapshot of Nixpkgs as a +dependency. This way, the complexity of shell scripting and calls to +language-specific tooling can be kept mostly hidden away from Nix packaging +expressions. * Frustrations acknowledged -Having covered so many of Nix's strengths, it's important to be aware of some +Having covered so many of Nix's strengths, it's good to be aware of some problems the Nix community is still working through. ** Nixpkgs takes time to learn There are parts of Nix that are notably simple. For example, there's an elegance -to the hashing calculation and how =/nix/store= is used. Furthermore the Nix -language itself has a small footprint, which eases learning it. +to the hashing calculation and how =/nix/store= is used. Furthermore, the Nix +language has a small footprint, making learning Nix easier. -However, because of the complexity of all the programming language ecosystems -out there, there are a /lot/ of supporting libraries in Nixpkgs to understand. -There's over two million lines of Nix in Nixpkgs, some auto-generated, -increasing the odds of getting lost in it. +However, because of the complexity of all the programming language ecosystems, +there are a /lot/ of supporting libraries in Nixpkgs to understand. There are +over two million lines of Nix in Nixpkgs, some auto-generated, increasing the +odds of getting lost. The [[nixpkgs-manual][official Nixpkgs manual]] only seems to cover a fraction of what package -authors really need to know. Invariably, people seem to master Nix by exploring -the source code of Nixpkgs, supplemented by example projects for reference. You -can get surprisingly far mimicking code you find in Nixpkgs that packages -something similar to what you have in front of you. But understanding what's -actually going on so you avoid simple mistakes can take some time. +authors need to know. Invariably, people seem to master Nix by exploring the +source code of Nixpkgs, supplemented by example projects for reference. You can +get surprisingly far mimicking code you find in Nixpkgs that packages something +similar to what you have in front of you. But understanding what's going on so +you avoid simple mistakes can take some time. Various people have attempted to fill the gap with documentation and tutorials. Even this document you're reading now is one such attempt. However, we're -missing searchable index of all the critical functions in Nixpkgs for people to -explore. Something as simple as parsed [[docstring][docstrings]] as an extension of the Nix +missing a searchable index of all the critical functions in Nixpkgs for people +to explore. Something as simple as parsed [[docstring][docstrings]] as an extension of the Nix language would go a long way, which would be far easier to implement than -something more involved like a type system for the Nix language. +something more involved, like a type system for the Nix language. ** Confusion of stability @@ -240,51 +238,54 @@ The Nix community seems divided into the following camps: - those who want new features and fixes to known grievances - those who want stable systems based on Nix in industrial settings. -It's not necessary for these groups to be at odds. Unfortunately, Nix has -released new experimental features in a way that has created confusion of how to -build stable systems with Nix. +These groups don't need to be at odds. Unfortunately, Nix has released +experimental features in a way that has created confusion about how to build +stable systems with Nix. *** Nix 2.0 and the new =nix= command An early complaint of Nix was the non-intuitiveness of Nix's original assortment -of command-line tools. To address this, Nix 2.0 introduced a unifying tool +of command-line tools. To address this, Nix 2.0 introduced a unifying CLI tool called =nix=. Despite appreciable improvements in user experience, the newer -=nix= command has taken some time for it to get enough functionality to actually -replace the older tools (=nix-build=, =nix-shell=, =nix-store=, etc.). For a -while, it's ended up yet another tool to learn. +=nix= command has taken some time to get enough functionality to replace the +older tools (=nix-build=, =nix-shell=, =nix-store=, etc.). For a while, it's +ended up yet another tool to learn. -If you look at the manpage for =nix= there's a clear warning at the top: +If you look at the manpage for the latest release of =nix=, there's a clear +warning at the top: #+BEGIN_QUOTE -Warning: This program is experimental and its interface is subject to change. +Warning: This program is experimental, and its interface is subject to change. #+END_QUOTE -This warning has been there since 2018 when Nix 2.0 released. +This warning has been there since 2018, when Nix 2.0 was released. However, =nix repl= is the only way to get to a [[repl][REPL session]] in Nix, which is an important tool for any programming language. The previous tool providing a REPL (=nix-repl=) has been removed from Nixpkgs. -This means that technically, the community is strongly encouraging, if not -forcing, users to use an experimental tool and providing little guidance on how -to use Nix with some assurance of stability. This is important for industrial -users who script solutions against the =nix= command-line tools. +Because something as basic as the REPL is only available with an experimental +feature, the Nix community is confusing guidance on using Nix with some +stability. Eventually, with the release of Nix 2.4, experimental features were turned into flags that needed to be explicitly enabled by users. One of these flags was =nix-command=, which now gates users from any subcommand of =nix= beyond =nix -repl=. However, because so many users were already using the new =nix= command, -the experimental =nix-command= feature is enabled by default if no experimental +repl=. However, because so many users already use the new =nix= command, the +experimental =nix-command= feature is enabled by default if no experimental features have been configured explicitly. In other words, Nix ships with an experimental feature enabled by default. -This almost indicates that the new =nix= command isn't too unstable. Except, Nix -2.4 did indeed change the API of =nix= subcommands. +Enabling the new =nix= command by default almost indicates it isn't too +unstable. However, Nix 2.4 did indeed change the API of =nix= subcommands. +Industrial users scripting against =nix= had to figure out the appropriate +changes. In practice, the =nix= subcommands are relatively reliable. They are -well-written and functionally robust. But the core maintainers are reserving the -right to change input parameterization and output formatting. +well-written and functionally robust. However, the core maintainers reserve the +right to change input parameterization and output formatting without bumping a +major version number. They communicate this risk only with the warning atop the manpage, which most users have been training one another to ignore. @@ -292,92 +293,120 @@ users have been training one another to ignore. *** Flakes as an experiment Though Nix expressions have an incredible potential to be precise and -reproducible, there has always been some backdoors to break the reliability of +reproducible, there have always been some backdoors to break the reliability of builds. For example, Nix expressions have the potential to evaluate differently depending on the setting of some environment variables like =NIX_PATH=. -The motivation for these relaxations of determinism has been to have a quick way -to let personal computing users have a convenient way to manage their -environments. Some people are careful to avoid accidentally having -non-deterministic builds. Still, accidents have occurred frequently enough for -the community to want better. It's frustrating to have a broken build because -someone else set an environment variable incorrectly. +The motivation for these relaxations of determinism has been a quick way to let +personal computing users have a convenient way to manage their environments. +Some people are careful to avoid accidentally having non-deterministic builds. +Still, accidents have occurred frequently enough for the community to want +better. It's frustrating to have a broken build because someone else set an +environment variable incorrectly. -Nix 2.4 corrected for this by introducing an experimental feature called -/flakes/. Users still have a largely ergonomic way to manage their environments, -but builds are more strictly deterministic. Determinism is a large reason many -turn to Nix in the first place. A nice benefit of strictly enforced determinism -is the ability to cache evaluations of Nix expressions, which can be expensive -to compute. +Nix 2.4 corrected this by introducing an experimental feature called /flakes/. +Flakes provide an ergonomic way to manage build environments, with more +guarantees of determinism. A nice benefit of strictly enforced determinism is +the ability to cache evaluations of Nix expressions, which can be expensive to +compute. All this is generally good news. Flakes address problems that industrial users of Nix have long had to deal with. -However, flakes are an experimental feature that users need to explicitly -enable. Similar to the =nix= command, the inputs and outputs of flake-related -subcommands might change slightly. Such changes have already happened. +However, flakes are an experimental feature that users need to enable +explicitly. Similar to the =nix= command, across versions, the inputs and +outputs of flake-related subcommands might change slightly. Furthermore, the +hashes computed by flakes can change as well. Such changes have already +happened. On top of this, because flakes are experimental, documentation of flakes is fractured in the official documentation. It almost seems like the Nix developers are delaying proper documentation until there's a declaration of stability. A -preferred alternative would be developing documentation more concurrently with -the implementation, using the comprehensibility of the documentation to inform -the design of the software. Features that are too hard to explain expose good -opportunities for redesign. +preferred alternative would be developing documentation concurrently with the +implementation, using the documentation's comprehensibility to inform the +software's design. Good opportunities for redesign can be found in features that +prove difficult to explain. All this puts industrial Nix users in an annoying place. Not using flakes and -instead coaching coworkers and customers on how to use Nix safely +instead of coaching coworkers and customers on how to use Nix safely - increases the likelihood of defects as people make honest mistakes -- reduces the likelihood of adoption, because people get frustrated with poor +- reduces the likelihood of adoption because people get frustrated with poor ergonomics and difficulty understanding nuances and corner cases. -However, if industrial users move to flakes to address these problems we have +However, if industrial users move to flakes to address these problems, we have the following problems: -- we have to be ready for the flakes API to change as it's technically +- we have to be ready for the flakes API to change, as it's technically experimental -- we have to accept some added training hurdles since documentation of flakes is - tucked behind documentation of non-flakes usage. +- we have to accept some added training hurdles since the documentation of + flakes is tucked behind documentation of non-flake usage. + +** A few gaps in determinism + +Nix offers world-class build determinism, especially with flakes. But it's +important to understand that this determinism is not infallible. To date, no +build system can claim to provide flawless determinism. + +Known gaps involve corner cases like the following. Consider a hypothetical +compiler that can auto-detect that a build machine has many cores, and enables +an optimization upon detection incompatible with machines with fewer cores. +While Nix will generate different hashes if the platform architecture changes, +say from X86 to ARM, it will not consider a machine with many cores different +from one with fewer. So our example optimizing compiler could cause a +frustrating problem. A local build on a machine with few cores may work as +expected. But if a cache had a optimized build from a machine with many cores, +it would be pulled down for the same hash, as a substitute for a local build. +This optimization would lead to defects running on the wrong machine. + +Note that in general, we benefit from downloading and running packages built on +more powerful machines, and in almost all cases, the clever optimizations of +various compilers are portable. + +Lapses in determinism caused by Nix expressions in Nixpkgs are generally +considered defects and handled through GitHub issues. Some may argue that this +is the best that we can do. + +Most people will never encounter such corner cases in practice, but it's +important to understand the limitations of an otherwise extremely strong +guarantee of determinism. * Encouraging development with flakes This project encourages the development of Nix projects using flakes. The -benefits seem to outweigh the risks of instability. This is not a choice made +benefits seem to outweigh the risks of instability. This choice is not made lightly, and this document is an exercise of due diligence to inform users of compromises. -Flakes are absolutely the future in Nix. They significantly address prior pains. -Furthermore, enough people across the world are using them that we have some -confidence that the Nix commands are reliable as implemented. The core -contributors have just been very slow to commit to the user interfaces and -experience. - -There might be documentation and training hurdles with flakes, but it's not -actually much better not using flakes. This is why this project includes -documentation and guides on Nix itself. - -It's also important to keep the risks of using experimental features in -perspective. Industrial users are more likely to script heavily against =nix= -commands than personal users. Upgrading anything risks small breaks to address. -For some industrial users, such breaks are insufferable in aggregate, even if -manageable individually. This leads to a desire to only use software that has -been officially released as stable and supported. Still, if you've read -documents like this one, and feel the history and risks have been well -explained, using flakes might be the best option, even with industrial scripting -that might break. - -Usage of flakes outside scripting has almost no risk at all. By calling =nix= -with a few extra arguments ~--extra-experimental-features 'nix-command flakes'~ -we can access flakes commands for single invocations, without needing to enable -flakes globally. You can even make an alias for your shell that might look like -the following: +Flakes are the future in Nix. They significantly address prior pains. +Furthermore, enough people worldwide are using them that we have some confidence +that the Nix commands are robust. + +Using Nix with flakes should lead to a mostly pleasant experience. There are +some things to look out for, though. + +** Limiting usage of experimental APIs + +If you write scripts that call =nix= commands or use flakes, they may break +slightly if you upgrade to a newer version of Nix. For example, the formatting +of standard output for a command might change. + +By calling =nix= with a few extra arguments ~--extra-experimental-features +'nix-command flakes'~ we can access flakes commands for single invocations +without enabling flakes globally. You can even make an alias for your shell that +might look like the following: #+begin_src sh :eval no alias nix-flakes = nix --extra-experimental-features 'nix-command flakes' #+end_src -This way there's less to type interactively. Just don't script against this -command, and there's no worry of scripts breaking if the flakes API changes. +This way, there's less to type interactively. Just don't script against this +command, so there's no worry of scripts breaking due to experimental features. + +** Keeping Nix version consistent + +You may find that you need to pin the version of Nix to the same version for all +your machines (because hashes could change between versions, which are saved in +=flake.lock= files). #+include: "internal/nix-introduction-include.org" diff --git a/doc/nix-language.org b/doc/nix-language.org index da94453..c6e9b40 100644 --- a/doc/nix-language.org +++ b/doc/nix-language.org @@ -13,9 +13,9 @@ rm --recursive --force nix_example This document is a quick introduction to the Nix programming language. You can use the [[nix][Nix]] command-line tools without understanding the programming -language (also called Nix). However, learning the Nix language, will allow you -to develop your own Nix packages and read the source code of others. A lot of -this code is in [[nixpkgs][Nixpkgs]], a centralized repository of Nix code for the entire Nix +language (also called Nix). However, learning the Nix language will allow you to +develop your own Nix packages and read the source code of others. A lot of this +code is in [[nixpkgs][Nixpkgs]], a centralized repository of Nix code for the entire Nix ecosystem. The Nix community has recently developed [[nix-language-tutorial][a tutorial for the language]], which may @@ -29,9 +29,9 @@ relative to other general-purpose programming languages. You can read this document without following along on your own computer. -If you do want to follow along, you need to [[file:nix-installation.org][install Nix]]. When installing Nix you -will need to have the =nix-command= experimental feature enabled to follow along -with this document. You won't need =flakes= enabled, though. +If you do want to follow along, you need to [[file:nix-installation.org][install Nix]]. When installing Nix, +you must enable the =nix-command= experimental feature to follow along with this +document. You won't need =flakes= enabled, though. * Primitive literals @@ -46,7 +46,7 @@ nix eval --expr '1 + 1' #+RESULTS: : 2 -Note, we just have to quote our entire expression for a shell invocation. For an +Note we have to quote our entire expression for a shell invocation. For an interactive session where this quoting isn't needed, you can use the =nix repl= command. @@ -110,8 +110,8 @@ nix eval --expr '"a" + "b"' Because of Nix's foundation as a “functional” programming language, you can't repeatedly bind values to variables as you may in other “imperative” languages. -When we bind a value to a name, it's permanently bound for the entire scope the -name exists within. We manage these scopes of bound names with +When we bind a value to a name, it's permanently bound for the entire scope +within which the name exists. We manage these scopes of bound names with /let-expressions/: #+begin_src sh :results output :exports both @@ -131,22 +131,22 @@ nix eval --expr 'let a = 1; a = 2; in a' 2>&1 || true #+end_src #+RESULTS: -: error: attribute 'a' already defined at (string):1:5 +: error: attribute 'a' already defined at «string»:1:5 : : at «string»:1:12: : -: 1| let a = 1; a = 2; in a +: 1| let a = 1; a = 2; : | ^ Note that semicolons are mandatory in all Nix forms that have them, including -let-expressions. Because of Nix's strict parsing you can neither elide -semicolons, nor put extra ones. +let-expressions. Because of Nix's strict parsing, you can neither elide +semicolons nor put extra ones. * String interpolation -Sometimes we build up small code snippets inline in a Nix expression, so it's +Sometimes, we build up small code snippets inline in a Nix expression, so it's useful to have string interpolation support. Similar to shell scripting, the -syntax for this follows: +syntax for this is as follows: #+begin_src sh :results output :exports both nix eval --expr ' @@ -158,7 +158,7 @@ nix eval --expr ' #+RESULTS: : "FooBar is a terrible name" -String interpolation is supported by both normal and multi-line strings. +Both simple and multi-line strings support string interpolation. You can only interpolate strings into strings. For instance, interpolating an integer won't work: @@ -170,13 +170,18 @@ nix eval --expr ' #+end_src #+RESULTS: -: error: cannot coerce an integer to a string -: -: at «string»:3:9: -: -: 2| let a_number = 42; -: 3| in "${a_number} is a terrible number" -: | ^ +#+begin_example +error: + … while evaluating a path segment + + at «string»:3:9: + + 2| let a_number = 42; + 3| in "${a_number} is a terrible number" + | ^ + + error: cannot coerce an integer to a string +#+end_example We can use a builtin =toString= function to coerce types to strings: @@ -189,17 +194,17 @@ nix eval --expr ' #+RESULTS: : "42 is a terrible number" -Note that unlike shell scripts, the curly braces are not optional for string -interpolation in Nix. This works out in our favor if we're writing shell scripts -inline in a Nix expression, because we can use ~$name~ for shell string +Note that, unlike shell scripts, the curly braces are not optional for string +interpolation in Nix. Curly braces can help when writing shell scripts inline +within a Nix expression because we can use ~$name~ for shell string interpolation and ~${nix_expr}~ for Nix string interpolation. If this is not -enough, within multiline strings, we can suppress interpolation with by using +enough, within multiline strings, we can suppress interpolation by using =''${…}= instead of just =${…}=. -This is a little tedious to illustrate in a shell example without dealing with -quote delimiting. In the following example, the shell will interpolate, but not -the final Nix expression because we have =''$= instead of just =$= in our final -string: +Illustrating this syntax in a shell example without dealing with +quote-delimiting. In the following example, =FROM_SHELL= is interpolated by the +shell, but =NOT_EXPANDED_BY_NIX= is not because we have =''$= instead of just +=$= in our final string: #+begin_src sh :results output :exports both FROM_SHELL='${NOT_EXPANDED_BY_NIX}' @@ -226,7 +231,7 @@ nix eval --expr "''In $FROM_SHELL expansion still happens.''" 2>&1 || true * Functions -Nix has first class functions. Nix's functions take in only one argument at a +Nix has first-class functions. Nix's functions take in only one argument at a time, and use a colon to separate the parameter name from the body of the function. Furthermore, Nix uses whitespace for function application: @@ -249,8 +254,9 @@ nix eval --expr '(a: b: a + b) 1 2' #+RESULTS: : 3 -In this case, when we apply ~1~ to ~a: b: a + b~, we get another function. When -we apply ~2~ to this resultant function, we finally get our answer ~3~. +In this case, we get another function when we apply the function ~a: b: a + b~ +to the argument ~1~. When we apply this resultant function to ~2~, we finally +get our answer ~3~. If you've heard of /currying a function/ in other languages with n-ary functions, you may recognize this technique. @@ -303,9 +309,9 @@ nix eval --expr '{ a = 1; b = 2; }.b' : { a = 1; b = 2; } : 2 -Note, =builtins= is just an attribute set that is in scope by default. And -=typeOf= is just an attribute that maps to a function that returns a string -indicating the type of the argument. +Note, =builtins= is just an attribute set in scope by default. And =typeOf= is +just an attribute that maps to a function that returns a string indicating the +type of the argument. Often used in Nix expressions, we can overlay sets on top of each other with the =//= operator: @@ -351,7 +357,7 @@ nix eval --expr '({ a, b }: a + b ) { a = 1; b = 2; }' #+RESULTS: : 3 -This basic pattern syntax is rigid, and we can't pass in a attribute set with +This basic pattern syntax is rigid, and we can't pass in an attribute set with attributes that don't match the pattern: #+begin_src sh :results output :exports both @@ -359,15 +365,25 @@ nix eval --expr '({ a }: a + 2 ) { a = 3; b = 4; }' 2>&1 || true #+end_src #+RESULTS: -: error: anonymous function at (string):1:2 called with unexpected argument 'b' -: -: at «string»:1:1: -: -: 1| ({ a }: a + 2 ) { a = 3; b = 4; } -: | ^ -: Did you mean a? +#+begin_example +error: + … from call site + + at «string»:1:1: + + 1| ({ a }: a + 2 ) { a = 3; b = 4; } + | ^ + + error: function 'anonymous lambda' called with unexpected argument 'b' + + at «string»:1:2: + + 1| ({ a }: a + 2 ) { a = 3; b = 4; } + | ^ + Did you mean a? +#+end_example -If we want to relax the destructuring to accept sets with other attributes we +If we want to relax the destructuring to accept sets with other attributes, we can use a “...” form: #+begin_src sh :results output :exports both @@ -378,7 +394,7 @@ nix eval --expr '({ a, ...}: a + 2 ) { a = 3; b = 4; }' : 5 When destructuring, we can still bind the whole set to a name if we want to -using a “@” form. +using an “@” form. #+begin_src sh :results output :exports both nix eval --expr '(s@{ a, b }: a + s.b ) { a = 2; b = 3; }' @@ -387,7 +403,7 @@ nix eval --expr '(s@{ a, b }: a + s.b ) { a = 2; b = 3; }' #+RESULTS: : 5 -Attribute sets also support an additional syntactic convenience when pulling in +Attribute sets also support an additional syntactic convenience when pulling locally bound values as attributes, which comes up a lot in Nix. For example, consider the way we're using ~a = a~ here: @@ -398,8 +414,8 @@ nix eval --expr 'let a = 3; in { a = a; }' #+RESULTS: : { a = 3; } -Rather than worrying about spelling the same name correctly both sides of -the ‘=’ for an attribute setting, we can use the =inherit= keyword: +Rather than worrying about spelling the same name correctly, both sides of the +‘=’ for an attribute setting, we can use the =inherit= keyword: #+begin_src sh :results output :exports both nix eval --expr 'let a = 3; in { inherit a; }' @@ -411,7 +427,7 @@ nix eval --expr 'let a = 3; in { inherit a; }' * Paths Because the Nix language was designed for building packages, file paths come up -frequently in Nix expressions. Nix conveniently has a /path/ type, indicated by +frequently in Nix expressions. Nix conveniently has a /path/ type indicated by identifiers with at least one slash: #+begin_src sh :results output :exports both @@ -427,29 +443,28 @@ nix eval --expr '/some/filepath' #+begin_quote *WARNING:* This section discusses a language feature of Nix that should be -avoided in Nix expressions. Use of this feature can lead to subtle build -breakages depending on how you've set the =NIX_PATH= environment variable. This -section is included only to explain the feature if you encounter it in other's -code. +avoided in Nix expressions. This feature can lead to subtle build breakages +depending on how you've set the =NIX_PATH= environment variable. This section +only explains the feature if you encounter it in someone else's code. #+end_quote -Up until now, all the Nix expressions we've seen have been purely deterministic. -=1 + 1= will always evaluate to =2=. This is a valuable property for a build -tool. If a Nix expression describes how to build a package, we want to build -it consistently every time. +Until now, all the Nix expressions we've seen have been purely deterministic. +=1 + 1= will always evaluate to =2=. Determinism is a valuable property for a +build tool. If a Nix expression describes how to build a package, we want to +build it consistently every time. -Unfortunately, Nix has a special environment variable =NIX_PATH= which can +Unfortunately, Nix has a special environment variable =NIX_PATH= that can provide mutable path references. For expressions that use the syntax described in this section, Nix expressions may reference paths that could change dynamically based on how =NIX_PATH= has been set. Builds relying on this are intrinsically non-deterministic. -=NIX_PATH= is a legacy environment variable, that the ecosystem is slowly -working to phase out. To assist with this, the =nix= command requires an -=--impure= switch to evaluate an expressions that access mutable paths. +=NIX_PATH= is a legacy environment variable the Nix ecosystem is slowly working +to phase out. As part of this phase-out, the =nix= command requires an +=--impure= switch to evaluate expressions that access mutable paths. As with =PATH=, the settings within =NIX_PATH= are colon-delimited, with earlier -settings taking precedence over later ones. There are two forms for setting +settings taking precedence over later ones. There are two forms of setting =NIX_PATH=: - ~=~ @@ -472,8 +487,7 @@ If we create some files or folders there: mkdir --parents /tmp/some.d/path #+end_src -Then we can access them using our name as a path prefix in our angle -brackets: +Then we can access them using our name as a path prefix in our angle brackets: #+begin_src sh :results output :exports both NIX_PATH=temporary=/tmp nix eval --impure --expr '' @@ -482,10 +496,10 @@ NIX_PATH=temporary=/tmp nix eval --impure --expr '' #+RESULTS: : /tmp/some.d/path -If we use the second form for =NIX_PATH=, can specify directories without a -name. These directories are then used as candidate prefixes until an existent -path if found. For example, we can consider =/tmp= as a path prefix when looking -up =some.d/path= to find =/tmp/some.d/path=: +You can specify directories without a name using the second form for =NIX_PATH=. +These directories are then used as candidate prefixes until an existent path is +found. For example, we can consider =/tmp= as a path prefix when looking up +=some.d/path= to find =/tmp/some.d/path=: #+begin_src sh :results output :exports both NIX_PATH=/tmp nix eval --impure --expr '' @@ -494,19 +508,19 @@ NIX_PATH=/tmp nix eval --impure --expr '' #+RESULTS: : /tmp/some.d/path -Now you know about the angle bracket syntax, but please never use it. It's +You now know about the angle bracket syntax, but please never use it. It's generally caused the Nix community grief. The Nix community looks bad when a build system advertising deterministic builds fails to do so. There's almost always a better way to accomplish what you might with mutable path references. * Other (mutable) dangers -There's other ways to have mutable references in Nix, but the angle notation +There are other ways to have mutable references in Nix, but the angle notation discussed in the prior section is the most common found across various projects and legacy documentation. -For instance, it's possible to use some functions found on the =builtins= set to -fetch files from the internet. Here's one such example: +For instance, it's possible to use some functions on the =builtins= set to fetch +files from the internet. Here's one such example: #+begin_src sh :eval no :exports code nix eval --impure --expr ' @@ -527,26 +541,26 @@ nix eval --impure --expr ' #+RESULTS: #+begin_example { - lastModified = 1678188176; - lastModifiedDate = "20230307112256"; - narHash = "sha256-IH80NcLhwjGpIXEjHuV+NgaSC+Y/PXquxZ/C8Bl+CLk="; - outPath = "/nix/store/4hm39fpkqz41jqpz21yf98mcisb1k9bx-source"; - rev = "ea2fca765c440fff1ff74e1463444dea7b819db2"; - revCount = 808; - shortRev = "ea2fca7"; + lastModified = 1705312282; + lastModifiedDate = "20240115095122"; + narHash = "sha256-PPXqKY2hJng4DBVE0I4xshv/vGLUskL7jl53roB8UdU="; + outPath = "/nix/store/wjb45dlgycjw8759q43js031yjn5l0g5-source"; + rev = "7c2f768bf9601268a4e71c2ebe91e2011918a70f"; + revCount = 843; + shortRev = "7c2f768"; submodules = false; } #+end_example -Notice that because we're referencing a URL on the internet that might change +Notice that because we're referencing a URL on the internet, possibly volatile, =nix eval= forces us to use the =--impure= switch to perform this evaluation. -In general, exercise caution any time calling a nix command with =--impure=. +In general, exercise caution when calling a nix command with =--impure=. * Importing -We can import paths. If the path is a file, it's loaded as a Nix expression. If -it's a directory, a file called “default.nix” is loaded within it. +We can import paths. If the path is a file, it's loaded as a Nix expression. +If it's a directory, a file called “default.nix” is loaded within it. The Nixpkgs source code, for example, has a =default.nix= file at its root, so we can import a path directly to it @@ -565,8 +579,8 @@ nix eval --file ./nix_example #+RESULTS: : 3 -We also see here use of the =--file= switch with =nix=. This is useful when an -expression is saved in a file. +Here we also see the usage of the =--file= switch with =nix=. This switch is +useful when an expression is saved in a file. * Org-mode Cleanup :noexport: diff --git a/doc/nix-usage-flakes.org b/doc/nix-usage-flakes.org index 49de855..ae92b86 100644 --- a/doc/nix-usage-flakes.org +++ b/doc/nix-usage-flakes.org @@ -3,7 +3,7 @@ * Org-mode setup :noexport: -This document is written in a project-agnostic way so it can be copied to other +This document is written in a project-agnostic way to be copied to other projects that use Nix. ** Variables @@ -26,7 +26,7 @@ following macros and source code blocks (using Noweb). #+macro: run-attr-long {{{get(run-attr-long,=,=)}}} #+macro: run-name {{{get(run-target-short,“,”)}}} #+macro: run-target-short {{{get(run-target-short,=,=)}}} -#+macro: run-target-long {{{get(run-target-long,=,=)}}} +#+macro: run-target-long {{{get(run-target-short,=bin/,=)}}} #+macro: nix-latest {{{get(nix-latest)}}} #+macro: nixos-latest {{{get(nixos-latest)}}} #+macro: platforms {{{get(platforms)}}} @@ -52,7 +52,7 @@ an evaluation of a source code block. ** Setup action -Next we perform some side-effects to set up the evaluation of the whole +Next, we perform some side effects to set up the evaluation of the whole document. #+name: cleanup @@ -65,8 +65,8 @@ rm --force /tmp/nix-profile* This document explains how to take advantage of software provided by Nix for people new to [[nix][the Nix package manager]]. This guide uses this project for -examples, but it focused on introducing general Nix usage, which applies to -other projects using Nix as well. +examples but focuses on introducing general Nix usage, which also applies to +other projects using Nix. #+include: "internal/nix-usage-flakes-include.org" @@ -74,9 +74,8 @@ other projects using Nix as well. This project uses Nix to download all necessary dependencies and build everything from source. In this regard, Nix is helpful as not just a package -manager, but also a build tool. Nix helps us get from raw source files to not -only built executables, but all the way to a Nix package, which we can install -with Nix if we like. +manager but also a build tool. Nix helps us get from raw source files to built +executables in a package we can install with Nix. Within this project, the various files with a ~.nix~ extension are Nix files, each of which contains an expression written in the [[nix-language-manual][Nix expression language]] used @@ -96,7 +95,7 @@ Otherwise, see the provided [[file:nix-installation.org][Nix installation and co not yet set Nix up. To continue following this usage guide, you will need Nix's experimental flakes -feature. You can enable this globally, or use an alias such as the following: +feature. You can enable this globally or use an alias such as the following: #+begin_src sh :eval no alias nix-flakes = nix --extra-experimental-features 'nix-command flakes' @@ -104,25 +103,25 @@ alias nix-flakes = nix --extra-experimental-features 'nix-command flakes' * Working with Nix -Though covering Nix comprehensively is beyond the scope of this document, we'll +Though comprehensively covering Nix is beyond the scope of this document, we'll go over a few commands illustrating some usage of Nix with this project. ** Referencing flake projects -Most of this document illustrates use of the =nix= command, which provides a +Most of this document illustrates usage of the =nix= command, which provides a number of subcommands and centralizes Nix usage. Many of the =nix= subcommands accept references to flake-enabled projects. A -flake is written as just a Nix expression saved in a file named =flake.nix=. -This file should be at the root of a project. We can reference both local and -remote flake projects. +flake is specified with a Nix expression saved in a file named =flake.nix=. This +file should be at the root of a project. We can reference both local and remote +flake projects. -Here's some common forms we can use to reference flake projects: +Here are some common forms we can use to reference flake projects: | Syntax | Location | |----------------------------------+----------------------------------------------------------------| | ~.~ | flake in the current directory | -| ~~ | flake in some other filepath (must have a slash) | +| ~~ | flake in some other file path (must have a slash) | | ~~ | reference to flake from the registry (see =nix registry list=) | | ~git+~ | latest flake in the default branch of a Git repository | | ~git+?ref=~ | latest flake in a branch of a Git repository | @@ -135,10 +134,10 @@ This table introduces an angle-bracket notation for syntactic forms with components that change with context. This notation is used throughout this document. -Referencing local flake projects is easy enough with filepaths. But the URL-like -notation for remote flake projects can get a touch verbose to type out. -Furthermore, some of these references are not fixed. For example, Git branches -point to different commits over time. +Referencing local flake projects is easy enough with file paths. But the +URL-like notation for remote flake projects can get verbose. Furthermore, some +of these references are not fixed. For example, Git branches point to different +commits over time. To manage flake references, Nix provides a flakes registry. Upon installation this registry is prepopulated with some global entries: @@ -151,32 +150,34 @@ nix registry list #+RESULTS: nix-registry-list #+begin_example … +global flake:nixos-homepage github:NixOS/nixos-homepage +global flake:nixos-search github:NixOS/nixos-search global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable -global flake:templates github:NixOS/templates +global flake:nur github:nix-community/NUR global flake:patchelf github:NixOS/patchelf global flake:poetry2nix github:nix-community/poetry2nix -global flake:nix-serve github:edolstra/nix-serve -global flake:nickel github:tweag/nickel -global flake:bundlers github:NixOS/bundlers global flake:pridefetch github:SpyHoodle/pridefetch -global flake:helix github:helix-editor/helix global flake:sops-nix github:Mic92/sops-nix +global flake:systems github:nix-systems/default +global flake:templates github:NixOS/templates #+end_example For example, rather than referencing the flake on the =nixpkgs-unstable= branch of the Nixpkgs GitHub repository with ~github:NixOS/nixpkgs/nixpkgs-unstable~, -we can just use the simple identifier ~nixpkgs~. +we can use the simpler identifier ~nixpkgs~. -If we want to point to a different branch, but still use an identifier from the -registry, we can by extending it with the branch. For example, the flakes +If we want to point to a different branch but still use an identifier from the +registry, we can do so by extending it with the branch. For example, the flakes identifier ~nixpkgs~ is the same as ~nixpkgs/nixpkgs-ustable~, but we can also use {{{get(nixos-latest,~nixpkgs/nixos-,~)}}} to override the branch and point to the NixOS {{{nixos-latest}}} release branch. -Note, registries have mutable references, but for some of these references Nix -knows how to repeatably rebuild the snapshot referenced. For example, when -referencing a GitHub repository via a registry reference, Nix will take note of -the commit ID of the snapshot retrieved. +Note that registries have mutable references, but Nix knows how to rebuild the +snapshot referenced for some of these references deterministically. For example, +when referencing a GitHub repository via a registry reference, Nix will take +note of the commit ID of the snapshot retrieved. Nix typically stores this +information required for reproducibility in a /lock file/ called =flake.lock= +adjacent to =flake.nix=. ** Inspecting flake outputs @@ -198,25 +199,25 @@ nix flake show . | ansifilter #+RESULTS: nix-flake-show #+begin_example -git+file:///home/tnks/src/shajra/nix-project +git+file:///home/shajra/src/nix-project ├───apps │ ├───aarch64-darwin … +├───flakeModules: unknown +├───legacyPackages │ ├───aarch64-darwin omitted (use '--legacy' to show) │ ├───x86_64-darwin omitted (use '--legacy' to show) │ └───x86_64-linux omitted (use '--legacy' to show) ├───lib: unknown -├───nixosConfigurations -├───nixosModules ├───overlays │ └───default: Nixpkgs overlay ├───packages │ ├───aarch64-darwin -│ │ ├───nix-scaffold: package 'nix-scaffold' -│ │ └───org2gfm: package 'org2gfm' +│ │ ├───nix-scaffold omitted (use '--all-systems' to show) +│ │ └───org2gfm omitted (use '--all-systems' to show) │ ├───x86_64-darwin -│ │ ├───nix-scaffold: package 'nix-scaffold' -│ │ └───org2gfm: package 'org2gfm' +│ │ ├───nix-scaffold omitted (use '--all-systems' to show) +│ │ └───org2gfm omitted (use '--all-systems' to show) │ └───x86_64-linux │ ├───nix-scaffold: package 'nix-scaffold' │ └───org2gfm: package 'org2gfm' @@ -224,13 +225,13 @@ git+file:///home/tnks/src/shajra/nix-project └───default: template: A starter project using shajra/nix-project. #+end_example -Flake outputs are a organized in a tree of /attributes/. References to paths of +Flake outputs are organized in a tree of /attributes/. References to paths of attributes are dot-delimited. There is a standard schema for the output -attribute tree of flake. It's permitted to have outputs outside this schema. +attribute tree of a flake. Nix permits outputs outside this schema. -This document mostly focuses on packages provided by the =packages= output -attribute. Notice that a flake provides packages for different (but often not -all) system architectures. +This document focuses on packages provided by the =packages= output attribute. +Notice that a flake offers packages for different (but rarely all) system +architectures. For commands like =nix flake show= that expect a flake reference as an argument, =.= is assumed as default if an argument isn't provided. So ~nix flake show~ is @@ -278,15 +279,15 @@ for the provided name: Which attributes are searched depends on the =nix= subcommand called. -For commands accepting installables as an argument, if none is provided, then +For commands accepting installables as an argument, if none are provided, then =.= is assumed. Nix will attempt to read a =flake.nix= file in the current -directory. If not found, Nix will continue searching parent directories -recursively to find a =flake.nix= file. +directory. If not found, Nix will recursively search parent directories to find +a =flake.nix= file. ** Searching flakes for packages We can use the =nix search= command to see what package derivations a flake -contains. For example from the root directory of this project, we can execute: +contains. For example, from the root directory of this project, we can execute: #+begin_src sh :eval no nix search . @@ -307,13 +308,12 @@ nix search . | ansifilter : * legacyPackages.x86_64-linux.ci If a flake has a lot of packages, you can pass regexes to prune down the search. -Returned values will match all the regexes provided. Also, we can search a -remote repository as well for packages to install. +Returned values will match all the regexes provided. -For example, Nixpkgs is a central repository for Nix providing several thousand -packages. We can search the “nixpkgs-unstable” branch of [[nixpkgs][Nixpkgs' GitHub -repository]] for packages that match both “gpu|opengl|accel” and “terminal” as -follows: +We can also search a remote repository for packages to install. For example, +Nixpkgs is a central repository for Nix, providing several thousand packages. We +can search the “nixpkgs-unstable” branch of [[nixpkgs][Nixpkgs' GitHub repository]] for +packages that match both “gpu|opengl|accel” and “terminal” as follows: #+name: nix-search-remote-verbose #+begin_src sh :results output silent :exports code @@ -326,7 +326,7 @@ some typing: #+begin_src sh :eval no nix search nixpkgs 'gpu|opengl|accel' terminal -#+end_src ++end_src #+name: nix-search-remote-concise #+begin_src sh :dir .. :results output :exports results @@ -335,21 +335,24 @@ nix search nixpkgs 'gpu|opengl|accel' terminal | ansifilter #+RESULTS: nix-search-remote-concise #+begin_example -,* legacyPackages.x86_64-linux.alacritty (0.11.0) +,* legacyPackages.x86_64-linux.alacritty (0.13.1) A cross-platform, GPU-accelerated terminal emulator -,* legacyPackages.x86_64-linux.darktile (0.0.10) +,* legacyPackages.x86_64-linux.darktile (0.0.11) A GPU rendered terminal emulator designed for tiling window managers -,* legacyPackages.x86_64-linux.kitty (0.27.1) +,* legacyPackages.x86_64-linux.kitty (0.32.0) A modern, hackable, featureful, OpenGL based terminal emulator -,* legacyPackages.x86_64-linux.wezterm (20221119-145034-49b9839f) +,* legacyPackages.x86_64-linux.rio (0.0.34) + A hardware-accelerated GPU terminal emulator powered by WebGPU + +,* legacyPackages.x86_64-linux.wezterm (20230712-072601-f4abf8fd) GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust #+end_example -If we're curious what version of WezTerm is available in NixOS's latest release, -we can specialize the installable we're searching as follows: +If we're curious about what version of WezTerm is available in NixOS's latest +release, we can specialize the installable we're searching as follows: #+begin_src sh :eval no :noweb yes nix search nixpkgs/nixos-<>#wezterm @@ -361,21 +364,21 @@ nix search nixpkgs/nixos-<>#wezterm | ansifilter #+end_src #+RESULTS: nix-search-remote-wezterm -: * legacyPackages.x86_64-linux.wezterm (20220905-102802-7d4b8249) -: A GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust +: * legacyPackages.x86_64-linux.wezterm (20230712-072601-f4abf8fd) +: GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust -Here {{{get(nixos-latest,~/nixos-,~)}}} overrides the default =nixpkgs-unstable= branch of -the registry entry, and the ~#wezterm~ suffix searches not just the flake, but a -specific package named ~wezterm~, which will either be found or not (there's no -need for regexes to filter further). +Here {{{get(nixos-latest,~/nixos-,~)}}} overrides the default =nixpkgs-unstable= +branch of the registry entry, and the ~#wezterm~ suffix searches not just the +flake, but a specific package named ~wezterm~, which will either be found or not +(there's no need for regexes to filter further). You may also notice that the Nixpkgs flake outputs packages under the =legacyPackages= attribute instead of the =packages=. The primary difference is that packages are flatly organized under =packages=, while =legacyPackages= can be an arbitrary tree. =legacyPackages= exists specifically for the Nixpkgs -project, a central project to the Nix ecosystem that's existed long before -flakes. Beyond Nixpkgs, don't worry about =legacyPackages=. Packages from all -other flakes should generally be found under =packages=. +project, a central project to the Nix ecosystem that has existed long before +flakes. Beyond Nixpkgs, you don't have to think much about =legacyPackages=. +Packages from all other flakes should generally be found under =packages=. ** Building installables @@ -402,15 +405,15 @@ We can build this package with =nix build= from the project root: nix build .#<> #+end_src -The positional arguments to =nix build= are /installables/ as discussed in prior -sections. Here, the =.= indicates that our flake should be found from the -current directory. Within this flake we look for a package with an attribute +As discussed in prior sections, the positional arguments to =nix build= are +/installables/. Here, the =.= indicates that our flake should be found from the +current directory. Within this flake, we look for a package with an attribute name of {{{package-attr-short}}}. We didn't have to use the full attribute path {{{package-attr-long}}} because =nix build= will automatically look in the =packages= attribute for the system it detects we're on. -If we omit the attribute path of our installable, Nix try to build a default -package which it expects to find under the flake's =packages..default=. +If we omit the attribute path of our installable, Nix tries to build a default +package, which it expects to find under the flake's =packages..default=. For example, if we ran just ~nix build .~, Nix would expect to find a =flake.nix= in the current directory with an output providing a ~packages..default~ attribute with a package to build. @@ -427,7 +430,7 @@ found there. Once a package is built, its content in =/nix/store= is read-only (until the package is garbage collected, discussed later). After a successful call of =nix build=, you'll see one or more symlinks for each -package requested in the current working directory. These symlinks by default +package requested in the current working directory. These symlinks, by default, have a name prefixed with “result” and point back to the respective build in =/nix/store=: @@ -437,7 +440,7 @@ readlink result* #+end_src #+RESULTS: nix-build-link -: /nix/store/0lbyvrsgxdwmjkgnszx9hggjmd8lanqd-org2gfm +: /nix/store/c9pxlzsvv5rb40hd21cvhbab825ddcpj-org2gfm Following these symlinks, we can see the files the project provides: @@ -451,7 +454,7 @@ tree -l result* : └── bin : └── org2gfm : -: 1 directory, 1 file +: 2 directories, 1 file It's common to configure these “result” symlinks as ignored in source control tools (for instance, for Git within a =.gitignore= file). @@ -466,17 +469,17 @@ nix path-info .#<> #+end_src #+results: nix-build-path -: /nix/store/0lbyvrsgxdwmjkgnszx9hggjmd8lanqd-org2gfm +: /nix/store/c9pxlzsvv5rb40hd21cvhbab825ddcpj-org2gfm ** Running commands in a shell -We can run commands in Nix-curated environments with =nix shell=. Nix will take +We can run commands in Nix-curated environments with =nix shell=. Nix will take executables found in packages, put them in an environment's =PATH=, and then execute a user-specified command. -With =nix shell=, you don't even have to build the package first with =nix -build= or mess around with “result” symlinks. =nix shell= will build any -necessary packages required. +With =nix shell=, you don't have to build the package first with =nix build= or +mess around with “result” symlinks. =nix shell= will build any necessary +packages required. For example, to get the help message for the {{{run-target-short}}} executable provided by the package selected by the {{{run-attr-short}}} attribute path @@ -502,10 +505,11 @@ arguments to select packages to put on the =PATH=. The command to run within the shell is specified after the =--command= switch. =nix shell= runs the command in a shell set up with a =PATH= environment -variable including all the =bin= directories provided by the selected packages. +variable that includes all the =bin= directories provided by the selected +packages. -If you just want to enter an interactive shell with the set up =PATH=, you can -drop the =--command= switch and following arguments. +If you only want to enter an interactive shell with the configured =PATH=, you +can drop the =--command= switch and following arguments. =nix shell= also supports an =--ignore-environment= flag that restricts =PATH= to only packages selected, rather than extending the =PATH= of the caller's @@ -513,7 +517,7 @@ environment. With =--ignore-environment=, the invocation is more sandboxed. As with =nix build=, =nix shell= will select default packages for any installable that is only a flake reference. If no installable is provided to -=nix shell=, the invocation will look for the default package in under the +=nix shell=, the invocation will look for the default package under the =packages..default= attribute output by a flake assumed to be in the current directory. So, the following invocations are all equivalent: - ~nix shell~ @@ -526,15 +530,15 @@ The =nix run= command allows us to run executables from packages with a more concise syntax than =nix shell= with a =--command= switch. The main difference from =nix shell= is that =nix run= detects which executable -to run from a package. If we want something other than what can be detected, -then we have to continuing using =nix shell= with =--command=. +to run from a package. If we want something other than what can be detected, we +must continue using =nix shell= with =--command=. As with other =nix= subcommands, =nix run= accepts an installable as an argument -(but only one). If none if provided, then =.= is assumed. +(but only one). If none is provided, then =.= is assumed. If the provided installable is only a flake reference with no package selected, -then =nix run= searches the following flake output attribute paths in order for -something to run: +then =nix run= searches the following flake output attribute paths, in order, +for something to run: - =apps..default= - =packages..default= @@ -581,13 +585,14 @@ nix search --json .#<> | jq . #+RESULTS: nix-search-details : { : "packages.x86_64-linux.org2gfm": { +: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)", : "pname": "org2gfm", -: "version": "", -: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)" +: "version": "" : … -In the JSON above, the “pname” field indicates the package's name. In practice, -this may or may not differ from flake output name of the installable. +The “pname” field in the JSON above indicates the package's name. In practice, +the package's name may or may not differ from flake output name of the +installable. =nix run= works because the package selected by the output attribute name {{{run-attr-short}}} selects a package with a package name {{{run-name}}} that @@ -600,9 +605,9 @@ using =nix shell= with =--command=. In the examples above, we've used selected packages from this project's flake, like {{{get(run-attr-short,=.#,=)}}}. But one benefit of flakes is that we can -refer to remote flakes just as easily, like =nixpkgs#hello=. This means we can -build quickly build environments with =nix shell= or run commands with =nix run= -without committing to install software. +refer to remote flakes just as easily, like =nixpkgs#hello=. Referencing remote +flakes helps us quickly build environments with =nix shell= or run commands with +=nix run= without committing to install software. Here's a small example. @@ -614,7 +619,7 @@ nix run nixpkgs#hello #+RESULTS: nix-run-remote : Hello, world! -When using =nix shell=, we can even mix local flake reference with remote ones, +When using =nix shell=, we can even mix local flake references with remote ones, all in the same invocation: #+name: nix-shell-remote @@ -622,27 +627,26 @@ all in the same invocation: nix shell --ignore-environment \ nixpkgs#which \ .#<> \ - --command which <> + --command which <> #+end_src #+RESULTS: nix-shell-remote -: /nix/store/0lbyvrsgxdwmjkgnszx9hggjmd8lanqd-org2gfm/bin/org2gfm +: /nix/store/c9pxlzsvv5rb40hd21cvhbab825ddcpj-org2gfm/bin/org2gfm -This is all a consequence of everything discussed in previous sections, but it's -good to see clearly that what we do with local flake references can work just as -well with remote flake references. +What we do with local flake references can work just as well with remote flake +references. ** Installing and uninstalling programs -We've seen that we can build programs with =nix build= and then execute them -using the “result” symlink (=result/bin/*=). Additionally, we've seen that you -can run programs with =nix shell= and =nix run=. But these additional steps and -switches/arguments can feel extraneous. It would be nice if we could just have -the programs on our =PATH=. This is what =nix profile= is for. +We've seen that we can build programs with =nix build= and execute them using +the “result” symlink (=result/bin/*=). Additionally, we've seen that you can run +programs with =nix shell= and =nix run=. But these additional steps and +switches/arguments can feel extraneous. It would be nice to have the programs on +our =PATH=. This is what =nix profile= is for. -=nix profile= maintains a symlink tree, called a /profile/, of installed -programs. The default profile is at =~/.nix-profile=. For non-root users, if -this doesn't exist, =nix profile= will create it as a symlink pointing to +=nix profile= maintains a symlink tree of installed programs called a /profile/. +The default profile is at =~/.nix-profile=. For non-root users, if this doesn't +exist, =nix profile= will create it as a symlink pointing to =/nix/var/nix/profiles/per-user/$USER/profile=. But you can point =nix profile= to another profile at any writable location with the =--profile= switch. @@ -670,26 +674,25 @@ nix profile list #+name: nix-profile-list #+begin_src sh :dir .. :results output :exports results -nix profile list --profile /tmp/nix-profile +nix profile list --profile /tmp/nix-profile | ansifilter #+end_src #+RESULTS: nix-profile-list -: 0 git+file:///home/tnks/src/shajra/nix-project#packages.x86_64-linux.org2gfm git+file:///home/tnks/src/shajra/nix-project#packages.x86_64-linux.org2gfm /nix/store/0lbyvrsgxdwmjkgnszx9hggjmd8lanqd-org2gfm - -The output of =nix profile list= is a bit verbose, but each line has three parts: -- an index to use with other =nix profile= subcommands (like =nix profile remove=) -- the specified installable reference -- the resolved reference actually installed -- the store path in =/nix/store= +: Index: 0 +: Flake attribute: packages.x86_64-linux.org2gfm +: Original flake URL: git+file:///home/shajra/src/nix-project +: Locked flake URL: git+file:///home/shajra/src/nix-project +: Store paths: /nix/store/c9pxlzsvv5rb40hd21cvhbab825ddcpj-org2gfm -And if we want to uninstall a program from our profile, we do so by the index +If we want to uninstall a program from our profile, we do so by the index from +this list: #+name: nix-profile-remove #+begin_src sh :eval no :noweb yes nix profile remove 0 #+end_src -we can also provide a regex matching the full attribute path of the flake: +We can also provide a regex matching the full attribute path of the flake: #+begin_src sh :eval no :noweb yes nix profile remove '.*<>' @@ -701,8 +704,8 @@ nix profile remove --profile /tmp/nix-profile '.*<>' #+end_src Also, if you look at the symlink-resolved location for your profile, you'll see -that Nix retains the symlink trees of previous generations of your profile. In -fact you can even rollback to a previous profile with the =nix profile rollback= +that Nix retains the symlink trees of previous generations of your profile. You +can even roll back to an earlier profile with the =nix profile rollback= subcommand. You can delete old generations of your profile with the =nix profile wipe-history= subcommand. @@ -718,11 +721,12 @@ be removed by =nix store gc= can be found by starting with symlinks stored as - =/nix/var/nix/manifests=. For each package, Nix is aware of all files that reference back to other -packages in =/nix/store=, whether in text files or binaries. This helps Nix -assure that dependencies of packages reachable from GC roots won't be deleted. +packages in =/nix/store=, whether in text files or binaries. This dependency +tracking helps Nix ensure that dependencies of packages reachable from GC roots +won't be deleted by garbage collection. Each “result” symlink created by a =nix build= invocation has a symlink in -=/nix/var/nix/gcroots/auto= pointing back it. So we've got symlinks in +=/nix/var/nix/gcroots/auto= pointing back to it. So we've got symlinks in =/nix/var/nix/gcroots/auto= pointing to “result” symlinks in our projects, which then reference the actual built project in =/nix/store=. These chains of symlinks prevent packages built by =nix build= from being garbage collected. @@ -732,9 +736,9 @@ delete the “result” symlink created before calling =nix store gc=. Breaking symlink chains under =/nix/var/nix/gcroots= removes protection from garbage collection. =nix store gc= will clean up broken symlinks when it runs. -Note that everything under =/nix/var/nix/profiles= is considered a GC root as -well. This is why users by convention use this location to store their Nix -profiles with =nix profile=. +Everything under =/nix/var/nix/profiles= is also considered a GC root. This is +why users, by convention, use this location to store their Nix profiles with +=nix profile=. Also, note if you delete a “result*” link and call =nix store gc=, though some garbage may be reclaimed, you may find that an old profile is keeping the @@ -746,7 +750,7 @@ any running processes. In the case of =nix run= no garbage collection root symlink is created under =/nix/var/nix/gcroots=, but while =nix run= is running =nix store gc= won't delete packages needed by the running command. However, once the =nix run= call exits, any packages pulled from a substituter or built -locally are candidates for deletion by =nix store gc=. If you called =nix run= +locally are candidates for deletion by =nix store gc=. If you called =nix run= again after garbage collecting, those packages may be pulled or built again. * Next steps @@ -764,7 +768,7 @@ respective man pages for more. We didn't cover much of [[nixpkgs][Nixpkgs]], the gigantic repository of community-curated Nix expressions. -The Nix ecosystem is vast. This project and documentation illustrates just a +The Nix ecosystem is vast. This project and documentation illustrate only a small sample of what Nix can do. * Org-mode teardown :noexport: diff --git a/doc/nix-usage-noflakes.org b/doc/nix-usage-noflakes.org index 36a6172..f8d8d7d 100644 --- a/doc/nix-usage-noflakes.org +++ b/doc/nix-usage-noflakes.org @@ -3,7 +3,7 @@ * Org-mode setup :noexport: -This document is written in a project-agnostic way so it can be copied to other +This document is written in a project-agnostic way to be copied to other projects that use Nix. ** Variables @@ -27,7 +27,7 @@ following macros and source code blocks (using Noweb). #+macro: run-attr-long {{{get(run-attr-long,=,=)}}} #+macro: run-name {{{get(run-target-short,“,”)}}} #+macro: run-target-short {{{get(run-target-short,=,=)}}} -#+macro: run-target-long {{{get(run-target-long,=,=)}}} +#+macro: run-target-long {{{get(run-target-short,=bin/,=)}}} #+macro: nixos-latest {{{get(nixos-latest)}}} #+macro: platforms {{{get(platforms)}}} @@ -57,7 +57,7 @@ an evaluation of a source code block. ** Setup action -Next we perform some side-effects to set up the evaluation of the whole +Next, we perform some side effects to set up the evaluation of the whole document. #+name: cleanup @@ -69,30 +69,31 @@ rm --force /tmp/nix-profile* * About this document This document explains how to take advantage of software provided by Nix for -people new to [[nix][the Nix package manager]]. This guide uses this project for -examples, but it focused on introducing general Nix usage, which applies to -other projects using Nix as well. +people new to [[nix][the Nix package manager]]. This guide uses this project for examples +but focuses on introducing general Nix usage, which applies to other projects +using Nix. This project supports a still-experimental feature of Nix called /flakes/, which this document shows users how to use _without_. [[file:nix-usage-flakes.org][Another document]] explains how to do everything illustrated in this document, but with flakes. #+begin_quote -*_NOTE:_* The decision to use an experimental feature such as flakes comes with -trade-offs. Please read the provided [[file:nix-introduction.org][supplemental documentation on Nix]] if you're -unfamiliar with flakes or trade-offs of using Nix's experimental features. +*_NOTE:_* If you're new to flakes, please read the provided [[file:nix-introduction.org][supplemental +introduction to Nix]] to understand the experimental nature of flakes and how it +may or may not affect you. Hopefully, you'll find these trade-offs acceptable to +take advantage of the improved experience flakes offer. #+end_quote -Although this document avoids requiring enabling the experimental flakes -feature, it does encourage some usage of the “nix-command” experimental feature. -This feature exposes a variety of subcommands on the =nix= command-line tool. -These subcommands have been in broad usage and are safe to use. However, as -still marked as experimental, their input parameters or output formats are -subject to change. Be aware when scripting against them. +Although this document avoids enabling the experimental flakes feature, it +encourages some usage of the “nix-command” experimental feature. This feature +exposes a variety of subcommands on the =nix= command-line tool. These +subcommands have been widely used and are considered safe. However, as still +marked as experimental, their input parameters or output formats are subject to +change. Be aware when scripting against them. -In general, this document only explains usage of experimental =nix= subcommands -when there exist no other alternatives, or when the alternatives are considered -worse for new users. +This document only uses these experimental =nix= subcommands when there exists +no other alternatives, or when the alternatives are considered worse for new +users. #+include: "nix-usage-flakes.org::*How this project uses Nix" @@ -112,27 +113,27 @@ feature enabled. * Working with Nix -Though covering Nix comprehensively is beyond the scope of this document, we'll +Though comprehensively covering Nix is beyond the scope of this document, we'll go over a few commands illustrating some usage of Nix with this project. ** Nix files -As mentioned, Nix expressions are written in the Nix programming language and -saved in files with a ~.nix~ extension. These /Nix files/ can be collocated with -the source they build and package, but this isn't necessary or always the case. -Some Nix files retrieve all the source or dependencies they need from the -internet. +As [[file:nix-introduction.org][the introduction]] mentions, Nix builds are specified by /Nix expressions/ +written in the Nix programming language and saved in files with a ~.nix~ +extension. These /Nix files/ can be collocated with the source they build and +package, but this isn't necessary or always the case. Some Nix files retrieve +all the source or dependencies they need from the internet. -Various Nix commands accept filepaths to Nix files as arguments. If a filepath +Various Nix commands accept file paths to Nix files as arguments. If a file path is a directory, a file named ~default.nix~ is referenced within. ** Inspecting this project for packages -This project has a =default.nix= file at its root. This is a Nix expression that -allows users to access this project's flake outputs (defined in =flake.nix=) -without having the experimental flakes feature enabled. Using the =default.nix= -file as opposed to using the flake directly comes at the cost of some extra time -evaluating the expression. +This project has a =default.nix= file at its root. This file contains a Nix +expression allowing users to access this project's flake outputs (defined in +=flake.nix=) without enabling the experimental flakes feature. Using the +=default.nix= file instead of using the flake directly comes at the cost of some +extra time evaluating the expression. The Nix expressions of projects often evaluate to /attribute/ trees of packages. We can select out these packages by traversing an /attribute path/. These @@ -151,9 +152,9 @@ If the Nix expression in this file evaluates to an attribute set (a map of attribute names to values), the attributes of this set are bound to variables within the REPL session. Nested attribute sets build up our attribute tree. -There's an experimental command that's useful to use, =nix search=. This command -is safe to call. Just be aware that if you script using it, the input -parameters/switches or output formatting might change with later releases. +Though experimental, the command =nix search= is safe and helpful. Just be +aware that input parameters/switches or output formatting might change with +later releases if you script using it. We can use an =--extra-experimental-features nix-command= switch to use an experimental feature with =nix= for a single call. Putting this all together, @@ -182,7 +183,7 @@ nix --experimental-features nix-command search --file . '' | ansifilter … #+end_example -If you have the =nix-command= feature disabled, and typing out ~nix +If you have disabled the =nix-command= feature, and typing out ~nix --extra-experimental-features nix-command~ is too verbose for your tastes, consider setting an alias in your shell such as the following: @@ -197,10 +198,10 @@ indicates to start at the root of the tree. Note, there are some projects for which =nix search= won't work. These projects require extra approaches to work with =nix search= that are beyond the scope of -this document. For these projects, you can still navigate its attribute tree -with =nix repl=. Or you can try to read the source code of the Nix expressions. +this document. You can still navigate these projects' attribute tree with =nix +repl=. Or you can try to read the source code of the Nix expressions. -We can filter search results down by supplying regexes as additional position +We can filter search results down by supplying regexes as an additional position parameters: #+begin_src sh :eval no :noweb yes @@ -235,13 +236,13 @@ nix --experimental-features \ #+begin_example { "legacyPackages.aarch64-darwin.ci": { + "description": "", "pname": "nix-project-ci", - "version": "", - "description": "" + "version": "" }, "legacyPackages.x86_64-darwin.ci": { + "description": "", "pname": "nix-project-ci", - "version": "", … #+end_example @@ -263,12 +264,12 @@ If you decide to eventually [[file:nix-usage-flakes.org][try out flakes]], you'l comfortably search all projects providing a =flake.nix= file, including Nixpkgs, without even having to clone Git repositories yourself. -Note, this document intentionally doesn't cover either the =nix-channel= command -or the =NIX_PATH= environment variable. Using either of these legacy features of -Nix leads systems to unnecessary unreliability, compromising the reasons to -advocate for Nix in the first place. If you really want to track and access -remote repositories, access them with an explicit checkout of a pinned -version/commit. +This document intentionally doesn't cover the =nix-channel= command or the +=NIX_PATH= environment variable. Using either of these legacy features of Nix +leads systems to unnecessary unreliability, compromising the reasons to advocate +for Nix in the first place. Flakes have a registry system that addresses this +problem. If you want to track and access remote repositories without flakes, +access them with an explicit checkout of a pinned version/commit. ** Building packages @@ -285,9 +286,9 @@ nix --experimental-features 'nix-command' \ #+RESULTS: nix-search-specific : { : "packages.x86_64-linux.org2gfm": { +: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)", : "pname": "org2gfm", -: "version": "", -: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)" +: "version": "" : } : } @@ -304,7 +305,7 @@ nix-build --attr <> . #+end_src #+RESULTS: nix-build -: /nix/store/0lbyvrsgxdwmjkgnszx9hggjmd8lanqd-org2gfm +: /nix/store/c9pxlzsvv5rb40hd21cvhbab825ddcpj-org2gfm If we omit the path to a Nix file, =nix-build= will try to build =default.nix= in the current directory. If we omit the =--attr= switch and argument, @@ -327,7 +328,7 @@ readlink result* #+end_src #+RESULTS: nix-build-link -: /nix/store/0lbyvrsgxdwmjkgnszx9hggjmd8lanqd-org2gfm +: /nix/store/c9pxlzsvv5rb40hd21cvhbab825ddcpj-org2gfm Following these symlinks, we can see the files the project provides: @@ -341,7 +342,7 @@ tree -l result* : └── bin : └── org2gfm : -: 1 directory, 1 file +: 2 directories, 1 file It's common to configure these “result” symlinks as ignored in source control tools (for instance, for Git within a =.gitignore= file). @@ -389,8 +390,8 @@ nix --experimental-features 'nix-command' \ : Uses Emacs to export Org-mode files to GitHub Flavored : … -Similar to other Nix commands, using in ~--file .~ tells =nix shell= to read a -Nix expression from =./default.nix=. The positional arguments when calling =nix +Like other Nix commands, using ~--file .~ tells =nix shell= to read a Nix +expression from =./default.nix=. The positional arguments when calling =nix shell= with =--file= are the attribute paths selecting packages to put on the =PATH=. @@ -399,9 +400,9 @@ working with a flake. The command to run within the shell is specified after the =--command= switch. =nix shell= runs the command in a shell set up with a =PATH= environment -variable including all the =bin= directories provided by the selected packages. +variable, including all the =bin= directories provided by the selected packages. -If you just want to enter a shell with the set up =PATH=, you can drop the +If you want to enter a shell with the set up =PATH=, you can drop the =--command= switch and following arguments. =nix shell= also supports an =--ignore-environment= flag that restricts =PATH= @@ -419,7 +420,7 @@ Different from what =nix shell= does, =nix run= detects which executable to run from a package. =nix run= assumes the specified package provides an executable with the same name as the package. -Remember, the package's /name/ is not the same as the /attribute/ use to select +Remember, the package's /name/ is not the same as the /attribute/ used to select a package. The name is package metadata not shown by the default output of =nix search=, but we can get to it by using =--json=: @@ -438,14 +439,16 @@ nix --experimental-features \ #+RESULTS: nix-search-run : { : "packages.x86_64-linux.org2gfm": { +: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)", : "pname": "org2gfm", -: "version": "", -: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)" +: "version": "" : } : } -In the JSON above, the “pname” field indicates the package's name. In practice, -this may or may not differ from the last attribute in the attribute path. +The “pname” field in the JSON above indicates the package's name. In practice, +this name may differ from the last attribute in the attribute path. As detailed +in =man nix3-run=, =nix run= may alternatively detect the executable to run from +a “name” or a “meta.mainProgram” field. Here's an example of calling =nix run= with this project: @@ -506,24 +509,24 @@ Downloads from URLs are cached. In case you feel the URL you've downloaded from has changed, use the =--refresh= switch with your invocation. Since we can only specify one =--file= switch, we can't make a shell with -packages from multiple Nix projects. This is something that is possible with -flakes enabled, discussed in the companion [[file:nix-usage-flakes.org][usage guide for flakes]]. +packages from multiple Nix projects. This is possible with flakes enabled, +discussed in the companion [[file:nix-usage-flakes.org][usage guide for flakes]]. -Note, the documentation in this project steers people away from =nix-shell=, +Note the documentation in this project steers people away from =nix-shell=, which provides some conveniences at the expense of compromising reproducible builds. Specifically, =nix-shell= reads from the =NIX_PATH= environment variable. Allowing an environment variable like =NIX_PATH= to affect build results has largely been deemed a mistake by the Nix community. Flakes provide -the convenience of =nix-shell= but with a better way of tracking mutable -references called /flake registries/. +the convenience of =nix-shell= but with better tracking mutable references +called /flake registries/. ** Installing and uninstalling programs -We've seen that we can build programs with =nix-build= and then execute them -using the “result” symlink (=result/bin/*=). Additionally, we've seen that you -can run programs with =nix shell= and =nix run=. But these additional steps and -switches/arguments can still feel extraneous. It would be nice if we could just -have the programs on our =PATH=. This is what =nix-env= is for. +We've seen that we can build programs with =nix-build= and execute them using +the “result” symlink (=result/bin/*=). Additionally, we've seen that you can run +programs with =nix shell= and =nix run=. But these additional steps and +switches/arguments can still feel extraneous. It would be nice to have the +programs on our =PATH=. This is what =nix-env= is for. =nix-env= maintains a symlink tree, called a /profile/, of installed programs. The active profile is pointed to by a symlink at =~/.nix-profile=. By default, @@ -554,6 +557,7 @@ nix-env --profile /tmp/nix-profile \ #+RESULTS: nix-env-install : installing 'org2gfm' +: building '/nix/store/hs9xz17vlb2m4qn6kxfmccgjq4jyrvqg-user-environment.drv'... We can see this installation by querying what's been installed: @@ -569,8 +573,8 @@ nix-env --profile /tmp/nix-profile --query #+RESULTS: nix-env-query-2 : org2gfm -Note that this name we see in the results of =nix-env= is the package name, and -not the attribute path we used to select out our packages. Sometimes these are +Note that this name we see in the results of =nix-env= is the package name, not +the attribute path we used to select our packages. Sometimes, they are congruent, but not always. We can see the package name of anything we install by using =nix search= with a @@ -591,9 +595,9 @@ nix --experimental-features nix-command \ #+RESULTS: nix-search-2 : { : "packages.x86_64-linux.org2gfm": { +: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)", : "pname": "org2gfm", -: "version": "", -: "description": "Script to export Org-mode files to GitHub Flavored Markdown (GFM)" +: "version": "" : } : } @@ -613,25 +617,26 @@ nix-env --profile /tmp/nix-profile \ #+RESULTS: nix-env-uninstall : uninstalling 'org2gfm' -Summarizing what we'e done, we've installed our package using its attribute path -({{{package-attr-long}}}) within the referenced Nix expression. But we uninstall -it using the package name ({{{package-name}}}), which may or may not be the same -as the attribute path. When a package is installed, Nix keeps no reference to -the expression that evaluated to the derivation of the installed package. The +Summarizing what we've done, we've installed our package using its attribute +path ({{{package-attr-long}}}) within the referenced Nix expression. But we +uninstall it using the package name ({{{package-name}}}), which may not be the +same as the attribute path. When a package is installed, Nix keeps no reference +to the expression evaluated to obtain the installed package's derivation. The attribute path is only relevant to this expression. In fact, two different expressions could evaluate to the same derivation, but use different attribute paths. This is why we uninstall packages by their package name. -Also, if you look at the resolved location for your profile, you'll see that Nix -retains the symlink trees of previous generations of your profile. In fact you -can even rollback to a previous profile with the =--rollback= switch. You can -delete old generations of your profile with the =--delete-generations= switch. +Also, if you look at the symlink-resolved location for your profile, you'll see +that Nix retains the symlink trees of previous generations of your profile. You +can even roll back to an earlier profile with the =--rollback= switch. You can +also delete old generations of your profile with the =--delete-generations= +switch. ** Garbage collection Every time you build a new version of your code, it's stored in =/nix/store=. There is a command called =nix-collect-garbage= that purges unneeded packages. -Programs that should not be removed by =nix-collect-garbage= can by found by +Programs that should not be removed by =nix-collect-garbage= can be found by starting with symlinks stored as /garbage collection (GC) roots/ under three locations: @@ -640,11 +645,12 @@ locations: - =/nix/var/nix/manifests=. For each package, Nix is aware of all files that reference back to other -packages in =/nix/store=, whether in text files or binaries. This helps Nix -assure that dependencies of packages reachable from GC roots won't be deleted. +packages in =/nix/store=, whether in text files or binaries. This dependency +tracking helps Nix ensure that dependencies of packages reachable from GC roots +won't be deleted by garbage collection. Each “result” symlink created by a =nix-build= invocation has a symlink in -=/nix/var/nix/gcroots/auto= pointing back it. So we've got symlinks in +=/nix/var/nix/gcroots/auto= pointing back to it. So we've got symlinks in =/nix/var/nix/gcroots/auto= pointing to “result” symlinks in our projects, which then reference the actual built project in =/nix/store=. These chains of symlinks prevent packages built by =nix-build= from being garbage collected. @@ -652,17 +658,16 @@ symlinks prevent packages built by =nix-build= from being garbage collected. If you want a package you've built with =nix-build= to be garbage collected, delete the “result” symlink created before calling =nix store gc=. Breaking symlink chains under =/nix/var/nix/gcroots= removes protection from garbage -collection. =nix store gc= will clean up broken symlinks when it runs. +collection. =nix-collect-garbage= will clean up broken symlinks when it runs. -Note that everything under =/nix/var/nix/profiles= is considered a GC root as -well. This is why users by convention use this location to store their Nix -profiles with =nix-env=. +Everything under =/nix/var/nix/profiles= is also considered a GC root. This is +why users, by convention, use this location to store their Nix profiles with +=nix-env=. -Also, note if you delete a “result*” link and call =nix-collect-garbage=, though -some garbage may be reclaimed, you may find that an old profile is keeping the -program alive. As a convenience, =nix-collect-garbage= has a =--delete-old= -switch that will delete these old profiles (it just calls ~nix-env ---delete-generations~ on your behalf). +Also, note that if you delete a “result*” link and call =nix-collect-garbage=, +though some garbage may be reclaimed, you may find that an old profile keeps the +program alive. Use =nix-collect-garbage='s =--delete-old= switch to delete old +profiles (it just calls ~nix-env --delete-generations~ on your behalf). It's also good to know that =nix-collect-garbage= won't delete packages referenced by any running processes. In the case of =nix run= no garbage @@ -690,7 +695,7 @@ respective man pages for more. We didn't cover much of [[nixpkgs][Nixpkgs]], the gigantic repository of community-curated Nix expressions. -The Nix ecosystem is vast. This project and documentation illustrates just a +The Nix ecosystem is vast. This project and documentation illustrate only a small sample of what Nix can do. * Org-mode teardown :noexport: