Skip to content

deepfire/nix-pills-compact-edition

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
.gitignore
.gitignore
 
 
Nix-Pills-Compact-Edition.org
Nix-Pills-Compact-Edition.org
 
 
README.org
README.org
 
 
amend.sh
amend.sh
 
 

Repository files navigation

About this document

Nix Pills Compact Edition is an on-going attempt to reformulate the excellent series of articles by Luca Bruno, known as Nix Pills, with the following differences in mind:

  • a more compact, outline-based format
  • an attempt at a more definition-based approach
  • ..while trying to give a more complete picture

Mostly I have been outright plagiarizing Luca through liberal cut’n’paste, so this is very much a derived work, and so, as usual – all the greatness is due to Luca, all the mistakes were added by me.

The pie-in-the-sky

  • import the rest of Nix Pills
  • review the imported material, try to distill further
  • review the distillation, use the outline to recombine things, breaking inter-/Pill/ boundaries
  • review, research and add Proposed Extras:
    • generic things that didn’t get into original Nix Pills
    • Haskell-specific things
    • Emacs-specific things

The preferred way to edit this document

..is through the excellent Org Mode: http://orgmode.org/ – a richer, but still light-weight, alternative to Markdown.

Index

Proposed extras

  • Generic:
    • callPackageWith – the way callPackage is actually implemented
    • callPackageWithScope
    • build-support/replace-dependency
    • build-support/autonix/
      • KDE only
        • mkPackage
          • resolveDeps
        • renameDeps
        • propagateDeps
        • userEnvDeps
        • extendDerivation
      • mostly KDE, but also: zed, tessel, airfield, nixui
        • nativeDeps
      • Haskell, KDE
        • overrideScope
      • overrideDerivation

        overrideDerivation drv f takes a derivation (i.e., the result of a call to the builtin function derivation) and returns a new derivation in which the attributes of the original are overriden according to the function f. The function f is called with the original derivation attributes.

        overrideDerivation allows certain “ad-hoc” customisation scenarios (e.g. in ~/.nixpkgs/config.nix). For instance, if you want to “patch” the derivation returned by a package function in Nixpkgs to build another version than what the function itself provides, you can do something like this:

        mySed = overrideDerivation pkgs.gnused (oldAttrs: {
  name    = "sed-4.2.2-pre";
  patches = [];
  src     = fetchurl {
    url    = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2;
    sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k";
  };
});

        For another application, see build-support/vm, where this function is used to build arbitrary derivations inside a QEMU virtual machine.

    • build-support/source-from-head-fun/
  • Haskell:
    • mkScope
    • ghcWithPackages wrapper
    • overrideCabal

Nix Pill #1: why you should give it a try

Motivation: problems with other package managers

  • not being purely functional
  • multiple versions are possible, but take a lot of effort
  • changing installation paths means breaking implicit assumptions about the file system hierarchy
  • containers, as a blunt solution, have their serious drawbacks
    • nearly useless without: orchestration, shared package cache
  • per-language sandbox-like solutions: virtualenv, cabal sandbox
    • scope that is too narrow – compiler, C libraries are not taken into account
    • a bunch of specific solutions with different flaws

Nix way: purely functional, universal solution at the right level of abstraction

  • no assumption about the global state of the system
  • system composed of elements (called derivations), that are explicitly derived from each other
  • global, hash-addressed, immutable /nix/store, storing derivation outputs – results of building packages from source, where, during build:
    • all gratuitious variables are set to their mathematical equivalent of zero (or identity, if you will)
      • time/date
      • environment
      • non-essential file system contents – the build is chrooted
    • all essential variables (called build inputs) count into the hash of the store elements, and are fixed forever:
      • build toolchain – compiler / linker / etc.
      • other build time dependencies – kernel headers, libc, libraries, make, automake, coreutils
      • runtime dependencies
  • ..which means that nearly identical subsets of packages can peacefully coexist along each other, with their only difference, for example, being the compiler they were built with
    • which means painless, entirely separate upgrades – at the cost of disc space, of course
  • (indirectly) composing PATH from elements stored in /nix/store

Nix way: immutability

  • Other package managers mutate global state by replacing packages in-place:
    • the library namespace granularity is mostly just the soname
    • new library gets used by all the current packages, linking to that soname, mostly available at a well-known, standard, shared, global location
  • In Nix, the namespace granularity for libraries (and other inputs, build or runtime) is its hash:
    • a new version of glibc becomes available at a location based on its hash, which will be known only to the packages that will be newly built with that particular derivation as a specific build input – passed as if by a function call
  • Ergo, the Nix equivalent of the traditional glibc “upgrade” is recompiling everything!
    • In practice, build farms provide you with the pre-built derivations, through an opt-out mechanism known as binary cache
    • Security updates is a separate story, a story of compromise with the principle of purity
  • Another problem is that it’s hard to compose at runtime applications that don’t have in mind a pure functional model, and that can’t be adapted to:
    • example: Firefox looks for the Flash plugin binary at a global (albeit per-user) path
      • which breaks the Nix model, entirely
      • the solution (known as wrapping) is to produce another Firefox derivation, that will at runtime feed unmodified Firefox the plugin path we desire, allowing us to put the Flash binary into /nix/store
  • Yet another question is upgrading data:
    • typically done in-place by other package managers, in best tradition of mutation
    • Nix makes it your own responsibility

Conclusion

  • Nix (manual) maximises both flexibility and reproducibility of system composition
  • NixOps (based on Nix, Nixpkgs and NixOS) makes cloud deployment easy, consistent and reliable, effectively deprecating all existing containment and orchestration solutions, such as Puppet, Vagrant, you name it
  • Weak spots are:
    • dynamic composition at runtime – solutions exist, but are one-off
    • massive rebuilds due to fundamental component upgrades (kernel, compilers, base libraries) – mostly made irrelevant by build farms
  • However, reality shows that Nix is an eminently livable environment, and progress in solving the above problems is being steadily made
  • Nixpkgs (search) is a completely new repository of all existing software
    • fresh concept
    • growing contribution
    • the current state is far beyond the experimental stage

Nix Pill #2: install on your running system

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-2-install-on-your-running.html

Download

Installation

  • /nix/store and a separate user, to isolate the store and build processes:
adduser nix
mkdir -m 0755 /nix && chown nix /nix
  • From now on, all the operations we do on the shell are done from this nix user:
su - nix
tar -xf nix-1.9-x86_64-linux.tar.bz2
cd nix-1.9-x86_64-linux
./install

*** INCOMPLETE ***

Nix Pill #3: enter the environment

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-3-enter-environment.html

*** INCOMPLETE ***

Nix Pill #4: the basics of the language

Introduction

Nix-repl

This chapter makes a heavy use of nix-repl. To install it, issue nix-env -i nix-repl.

CAVEAT: the nix-repl syntax is slightly different than nix syntax when it comes to assigning variables.

Value types

Simple types
  • integer
  • string
  • path
  • boolean
  • null
Complex types
  • list
  • attribute set
  • function – yes, a first-class value that can be passed to and returned from functions

Operators

Basic arithmetic:
  • +, -, * and integer division as builtins.div:
nix-repl> 1+3
4
nix-repl> builtins.div 6 3
2
  • CAVEAT: / is path concatenation instead: 
    nix-repl> 2/3
/home/nix/2/3
    • Nix parsed 2/3 as a relative path to the current directory.
    • Paths are parsed as long as there’s a slash.
    • Therefore to specify the current directory, use ./.
    • In addition, Nix also parses urls.
    • Not all urls or paths can be parsed this way. If a syntax error occurs, it’s still possible to fallback to plain strings.
  • NOTE: builtins.div is not being used in the whole of Nixpkgs repository, hence its second-class syntax status.
Boolean expressions
  • ||, &&, !
  • !=, ==
  • less used tests: <, >, >=, <=
Other operators

Identifiers

Dash (-) is allowed in identifiers:

nix-repl> a-b
error: undefined variable `a-b' at (string):1:1
nix-repl> a - b
error: undefined variable `a' at (string):1:1

Strings

String literals
..are enclosed by double-quotes (“), or two single-quotes (”), with \-based escaping:
nix-repl> "''foo''"
"''foo''"
nix-repl> ''"foo"''
"\"foo\""
nix-repl> "\"foo\""
"\"foo\""
  • String literal syntax provides means for interpolation of expressions within ${...}: 
    nix-repl> foo = "strval"
nix-repl> "$foo"
"$foo"
nix-repl> "${foo}"
"strval"
nix-repl> "${2+3}"
error: cannot coerce an integer to a string, at (string):1:2
    • NOTE: ignore the foo = “strval” assignment, it’s nix-repl-specific syntax.
  • Escaping ${...} within double-quoted string literals is done with the backslash. Within two single quotes, it’s done with '':
nix-repl> "\${foo}"
"${foo}"
nix-repl> ''test ''${foo} test''
"test ${foo} test"

Lists

List
an immutable sequence of expressions delimited by space (not comma):
nix-repl> [ 2 "foo" true (2+3) ]
[ 2 "foo" true 5 ]
  • Adding or removing elements from a list is only possible through production of a new list.

Attribute sets

Attribute set
a set of associations between keys and values, where:
  • keys can be either/identifiers/ or strings, for the cases when desired key names aren’t valid identifiers
  • values can be arbitrary Nix expressions
  • Example value: 
    nix-repl> s = { foo = "bar"; a-b = "baz"; "123" = "num"; }
nix-repl> s
{ 123 = "num"; a-b = "baz"; foo = "bar"; }
    • The output from nix-repl is wrong, you can’t write { 123 = “num”; } because 123 is not an identifier.
    • Semicolon (;) is required after every key-value assignment.
    • For those reading Nix expressions from Nixpkgs: do not confuse attribute sets (which are values) with argument sets used in function definitions (which are argument specifiers).
  • Accessing elements:
nix-repl> s.a-b
"baz"
nix-repl> s."123"
"num"
  • Defining recursive attribute sets:
    • Exhibit of the problem:
nix-repl> { a = 3; b = a+4; }
error: undefined variable `a' at (string):1:10
  • Problem statement – a isn’t in scope for b
  • Solution: INCOMPLETE: URL
nix-repl> rec { a= 3; b = a+4; }
{ a = 3; b = 7; }

If expression

nix-repl> a = 3
nix-repl> b = 4
nix-repl> if a > b then "yes" else "no"
"no"
  • Both then and else must be available – so the value of the expression is always defined.

Let expression

  • Introducing variables into scope:
nix-repl> let a = 3; b = 4; in a + b
7
  • ..with recursion:
nix-repl> let a = 4; b = a + 5; in b
9
  • Variable scopes compose..:
nix-repl> let a = 3; in let b = 4; in a + b
7
  • ..with shadowing:
nix-repl> let a = 3; in let a = 8; b = 4; in a + b
12

With expression

  • with allows “opening” attribute sets, binding names of its keys to their corresponding values:
nix-repl> longExpression = { a = 3; b = 4; "123" = 5; }
nix-repl> longExpression.a + longExpression.b
7
nix-repl> with longExpression; a + b
7
  • CAVEAT: only valid identifiers from the set keys will be included
nix-repl> let a = 10; in with longExpression; a + b + longExpression."123"
19
  • CAVEAT: if an identifier is bound in the outer scope and is also present in the attribute set of with, it will not be shadowed
nix-repl> let a = 10; in with longExpression; a + b
14
nix-repl> let a = 10; in with longExpression; longExpression.a + b
7

Laziness

Nix evaluates expressions only when needed. This allows easy definition of mutually referencing entities and efficient handling of large package repository definitions.

*** INCOMPLETE ***

Nix Pill #5: functions and imports

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-5-functions-and-imports.html

*** INCOMPLETE ***

Nix Pill #6: our first derivation

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-6-our-first-derivation.html

*** INCOMPLETE ***

Nix Pill #7: a working derivation

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-7-working-derivation.html

*** INCOMPLETE ***

Nix Pill #8: generic builders

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-8-generic-builders.html

*** INCOMPLETE ***

Nix Pill #9: automatic runtime dependencies

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-9-automatic-runtime.html

*** INCOMPLETE ***

Nix Pill #10: developing with nix-shell

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-10-developing-with-nix-shell.html

*** INCOMPLETE ***

Nix Pill #11: the garbage collector

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-11-garbage-collector.html

*** INCOMPLETE ***

Nix Pill #12: the inputs design pattern

Composing package definitions: repositories in Nix

  • We only packaged a single program so far – but how do we compose package definitions?
  • As we have already seen, from the point of view of a single package X, Nix is a language for describing:
    • names of the externalities that are required to build (and run) package X
    • how to use these externalities, given their names
    • ..which looks suspiciously like function definition – which it is!
  • However, function definition at package level isn’t enough for whole-system description:
    • ..functions need arguments supplied, which is the Nix way of saying that:
    • ..packages need to have their dependencies supplied
    • ..which means that particular versions and build configurations of the dependencies need to be decided upon somewhere
    • Traditionally, this somewhere is called a package repository
  • Nix, by itself, doesn’t enforce a package repository structure, as the only inherent requirement of its functional decomposition approach is that all the functions that define packages must be supplied proper arguments.
  • Nix, however has a particular package repository, with a particular structure – Nixpkgs
    • essentially, a single, giant expression in the Nix language:
      • mostly organized across individual, per-package files,
      • the root import-ing the nodes and leaves
      • evaluates to a giant attribute set with name -> package pairs
      • ..which works efficiently, due to the lazy evaluation property of Nix, meaning it only evaluates parts of the expression that are actually needed
    • ..which contrasts with, for example Debian and Fedora, which pull package definitions from several repositories (through indexes like /etc/apt/sources.list)
    • ..but coincides, for example, with Gentoo
  • The structure of Nixpkgs has patterns (like the above – pulling everything into a single coherent definition) that aren’t enforced by Nix, but are nonetheless present, codifying a distillation of successful practices of describing the world

Packaging graphviz

  • Graphviz:
  • Expression: 
    let
  pkgs = import <nixpkgs> {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
}
    • reuses autotools.nix from hello.nix
  • Build, producing runnable binaries under result/bin:
nix-build graphviz.nix
  • Let’s create a simple png: 
    $ echo 'graph test { a -- b }' | result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]
    • ..meaning that only the output formats graphviz supports natively, without using any extra library, were built.
  • in autotools.nix there’s a buildInputs variable, which gets concatenated to baseInputs. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.

Digression about gcc and ld wrappers

  • build systems for gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don’t use pkg-config to specify which flags to pass to the compiler, and so rely, instead, on the traditional, system-global locations, such as /usr/lib and /usr/include to find dependency headers and binaries – which are exactly absent in the Nix model.
  • gcc and binutils package definitions provided by Nixpkgs include wrappers, that allow passing extra arguments to gcc and ld binaries – bypassing and overriding the project build systems we call into, and effectively providing us with a project-independent way of supplying tool flags and dependencies:
    NIX_CFLAGS_COMPILE
    extra flags to gcc at compile time
    NIX_LDFLAGS
    extra flags to ld
  • These variables can be filled from derivation inputs the same way as was previously done for PATH – here is the relevant snippet of setup.sh: 
    for p in $baseInputs $buildInputs; do
  if [ -d $p/bin ]; then
    export PATH="$p/bin${PATH:+:}$PATH"
  fi
  if [ -d $p/include ]; then
    export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
  fi
  if [ -d $p/lib ]; then
    export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
  fi
done
    • The -rpath flag in ld is needed because at runtime, the executable must use exactly that version of the library.
    • If unneeded paths are specified, the fixup phase will automatically shrink the rpath.

Completing graphviz with gd

Building upon the results above, we now can transparently supply the graphviz build system with more libraries – which it will find without any configure parameters, thanks to the gcc and ld wrappers:

# graphviz.nix
let
  pkgs = import <nixpkgs> {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
  buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}

Composing package definitions: the repository expression

  • It’s nice to be able to abstract out the file-level repository structure, replacing the file paths with names, and that’s what Nixpkgs does – the top level expression imports the file names and provides the results as elements of the attribute set:
# default.nix:
{
  hello    = import ./hello.nix;
  graphviz = import ./graphviz.nix;
}
  • Trying it:
$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»
  • With nix-build: 
    $ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!
    • The -A argument is used to access an attribute of the set from the given .nix expression.
    • When a directory (by default the current directory) has a default.nix, it will be used by default, so the following will work as well:
nix-build -A hello
  • Install the package in your user environment: 
    $ nix-env -f . -iA graphviz
[...]
$ dot -V
    • -f is used to specify the expression to use, in this case the current directory, therefore ./default.nix.
    • -i stands for installation
    • -A is the same as above for nix-build
  • ..which concludes an exhibit of the essence of nixpkgs – a collection of package definitions.

The inputs pattern

  • Three problems with hello.nix and graphviz.nix definitions, rooting, essentially in their dependence on Nixpkgs structure:
    • They import Nixpkgs directly. In autotools.nix instead we pass Nixpkgs as an argument. That’s a much better approach.
    • No way to define graphviz without libgd support
    • No way to vary libgd version in graphviz definition
  • So far, the answer was to edit the callee
  • The essence of the inputs pattern is to actually use the functional abstraction, shifting these high-level decisions where they belong – to the caller
Inputs of an expression
the set of derivations needed to build that expression. In this case:
  • mkDerivation from autotools. Recall that mkDerivation has an implicit dependency on the toolchain.
  • libgd and its dependencies.
  • NOTE: src is also an input but it’s pointless to change the source from the caller. For version bumps, in Nixpkgs we prefer to write another expression (e.g. because patches are needed or different inputs are needed).

Leveraging functional abstraction for package expression independence

  • for graphviz.nix: 
    { mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:

mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
  buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}
    • {...}: ... is syntax for defining functions accepting an attribute set as argument. INCOMPLETE: URL
    • when omitted by the caller, gdSupport defaults to true
  • for default.nix: 
    let
  pkgs         = (import <nixpkgs>) {};
  mkDerivation = (import ./autotools.nix) pkgs;
in with pkgs; {
  hello        = (import ./hello.nix)    { inherit mkDerivation; };
  graphviz     = (import ./graphviz.nix) { inherit mkDerivation gd fontconfig libjpeg bzip2; };
  graphvizCore = (import ./graphviz.nix) { inherit mkDerivation gd fontconfig libjpeg bzip2;
                                           gdSupport = false; };
}
    • let binds convenience variables
    • for pedagogical purposes we cheat, by using a <nixpkgs>, which already contains everything one might want – defining contents of <nixpkgs> from ground up would have obscured the subject matter of this exhibition.
    • with pkgs “opens” the Nixpkgs, binding gd, fontconfig, libjpeg and bzip2
    • the toolchain is captured in the particular value of mkDerivation
    • inherit a b c; is syntactic sugar for a = a; b = b; c = c; – essentially capturing a part of the variable scope in an attribute set
    • import reads the .nix files, which contain functions – which are subsequently called with the attribute sets arguments – which is clarified by the added parentheses (which are otherwise unnecessary)

Conclusion

The inputs pattern is another name for leveraging functional abstraction to separate the repository in two conceptual parts:

Package definitions
import-ed leaves, containing flexible function expressions, that are free from policy decision-making, such as:
  • repository structure
  • specific versions of build inputs
  • other choices that the author of a particular package function expression chose to abstract out
High-level structure
the part that:
  • maintains knowledge of the file structure of repository, at the points of import expressions
  • provides a global namespace of package names, associating them to attribute sets, which are produced by:
    • instantiation of the import-ed package functions
      • note that the same package function can be instantiated several times, resulting in several packages, bound to different package names
    • ..while making decisions about their arguments – whose interpretation is determined by the package functions

Nix Pill #13: the callPackage design pattern

Origin: http://lethalman.blogspot.ru/2014/09/nix-pill-13-callpackage-design-pattern.html

The callPackage convenience

  • Problem: duplicate listing of package function arguments – explicit argument passing:
    • In package function:
# package1.nix
{ input1, input2, ... }:
...
  • In package derivation expressions:
rec {
  lib1     = import package1.nix { inherit input1 input2 ...; };
  program2 = import package2.nix { inherit inputX inputY lib1 ...; };
}
  • NOTE: this “problem” wouldn’t have arised, had we not strategically chosen the package function parameter names to coincide with the derivation names in the global derivation namespace. To be clear: this is a simplification opportunity, not a problem.
  • So, what form do we choose for the lucky derivation expressions – those enjoying the parameter/namespace coincidence? How about this:
{
  lib1     = callPackage package1.nix { };
  program2 = callPackage package2.nix { someoverride = overriddenDerivation; };
}
  • The exact semantics should this implicit argument passing have? This:
    • Import the given expression (which still must evaluate to a function).
    • Determine the name of its arguments.
    • Pass the values bound within global derivation namespace to the names of those arguments, and let us override those arguments.

Implementing callPackage

  • To implement implicit argument passing, we need a way to introspect at runtime the argument names of a function, to know what names it needs: 
    nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }
    • As a bonus, builtins.functionArgs returns not just the names, but also a boolean that determines whether the names have a default value or not.
  • We need access to the global derivation namespace, to know what is available
  • Given both of the above, the next step is to extract a subset from the global derivation namespace, determined by the chosen names: 
    # Mocking the /global derivation namespace/ here:
nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }
    • builtins.intersectAttrs is the function that does the extraction
  • This allows for a preliminary implementation of callPackage – function calling with implicit argument passing, without overrides: 
    nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set)
nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8
    • callPackage is a function of two arguments:
      • the attribute set from which to take arguments for..
      • ..the function that is to be called
    • the second expression shows the original function call expression, with explicit argument passing that we sought to avoid
  • What if the attribute set is missing a key/value pair for the required argument of the function being called? Nothing special – that’s an error.
  • The remaining piece is being able to override the attribute set, for example if the input derivation we want to supply is named with something else than the canonical derivation name – like gcc-5.2, instead of gcc.
  • This can be done through adding a third argument to callPackage – the override attribute set: 
    nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides)
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15
    • the // operator is an attribute set union, with the attribute set on the right taking precedence in case of key conflicts.

Use callPackage to simplify the repository

Given our brand new tool, we can simplify the repository expression (default.nix):

let
  nixpkgs     = import <nixpkgs> {};
  allPkgs     = nixpkgs // pkgs;
  callPackage = path: overrides:
    let f = import path;
    in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs)
          // overrides);
  pkgs        = with nixpkgs; {
    mkDerivation = import ./autotools.nix nixpkgs;
    hello        = callPackage ./hello.nix { };
    graphviz     = callPackage ./graphviz.nix { };
    graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
  };
in pkgs
  • We renamed the old pkgs of the previous pill to nixpkgs. “Our” package attribute set is now instead named pkgs.
  • For convenience, in callPackage we first import the file argument, instead of calling it directly. Otherwise each derivation expression would have to perform the import itself.
  • Since our expressions use packages from Nixpkgs, in callPackage we use allPkgs, which is the union of Nixpkgs and our packages.
  • We moved mkDerivation into pkgs itself, so that it gets also passed implicitly

Note how easy is to override arguments in the case of graphviz without gd. But most importantly, how easy it was to merge two repositories: nixpkgs and our pkgs!

NOTE: ..how deeply the new scheme depends on the ability of let to define mutually recursive structure – which is only made possible by lazy evaluation semantics of Nix:

  • allPkgs depends on pkgs
  • callPackage depends on allPkgs
  • pkgs depends on callPackage

Conclusion

  • Implicit argument passing enabled by callPackage, (aka the callPackage pattern) allow us to reduce maintenance burden:
    • whenever the set of package function arguments change, we’ll most likely need to change just one place – the package function itself
    • whenever the derivation expression needs an override of the default inputs to the package function, it can easily do just this
  • builtins.functionArgs is undocumented in the Nix Manual, mostly because it’s a function that only makes sense for the sort of plumbing we’re going through.
  • Most of all, we get to see how Nix, the language, is a generic tool suitable for construction of arbitrary purpose-built abstractions, that can support any policy we choose.

Nix Pill #14: the override design pattern

Prelude: about composability

  • Functional update pattern: update functions return a modified copy of the original structure.
Common type in Nix
a -> Derivation
  • Example:
    • input derivation drv
    • with debug info
    • with patches applied 
      debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)

The override pattern

Let us recall the repository structure:

A recursive attribute set
Name -> Derivation
  • ..that employs introspection to supply arguments – that are its own elements – to the Derivation expressions
  • ..it has the following form: 
    graphviz = callPackage ./graphviz.nix { };
  • If we wanted to produce a derivation of graphviz with a customized gd version, we would have to: 
    mygraphviz = callPackage ./graphviz.nix { gd = customgd; };
  • The problem here is a little unobvious – our override is tightly coupled to the original definition – which forces us to:
    1. find the original definition
    2. derive a modified version
    3. maintain the modification against the possible changes of the original
  • So, we would like to decouple the override from the form of the definition: 
    mygraphviz = graphviz.override { gd = customgd; };
    • NOTE: in this form, .override is just a plain element of an attribute set

The override implementation

As a reminder, the graphviz attribute is the derivation (which is an attribute set) returned by the derivation function import-ed from graphviz.nix.

So, to follow the form we preferred above, we would have to modify the import-ed function to return its original attribute set enriched with an element named override.

  • First attempt: 
    {
  # makeOverridable :: (ASet Inputs -> Drv) -> ASet Inputs
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: f (origArgs // newArgs); };
}
    • takes a derivation function
    • returns a derivation function, that returns a derivation with an enriched attribute set – the added override attribute
    • the override attribute too is a derivation function, but one, that calls the original derivation function with original arguments enriched by the attribute set supplied to the override function
  • How does it work? 
    $ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
    • so far, so good..
  • But, can we chain overrides? 
    nix-repl> res.override { a = 10; }
{ result = 15; }
    • ..the result of an override is missing the override attribute, so no, we can’t.
  • Try #2: 
    rec {
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}
    • the rec keyword allows us to refer to makeOverridable from its own definition
    • overrides can now be chained

Conclusion

  • makeOverridable allows us to decouple expressions defining derivations, from the points where we want to override them.
  • This allows a degree of separation between the package definitions, and the package users.

Nix Pill #15: nix search paths

Origin: http://lethalman.blogspot.ru/2014/09/nix-pill-15-nix-search-paths.html

*** INCOMPLETE ***

Nix Pill #16: nixpkgs, the parameters

Origin: http://lethalman.blogspot.ru/2014/11/nix-pill-16-nixpkgs-parameters.html

*** INCOMPLETE ***

Nix Pill #17: nixpkgs, overriding packages

Origin: http://lethalman.blogspot.ru/2014/11/nix-pill-17-nixpkgs-overriding-packages.html

Overriding a package

  • In Nix Pill #14, to produce an overriden derivation by producing a modified derivation expression, in order to avoid coupling we have built-in overridability into the derivation function, by putting an override function into the attribute set returned by the derivation function under the name override.
  • As an example usage, here’s how we override graphviz to build without X support: 
    $ nix-repl
nix-repl> :l <nixpkgs>
Added 4360 variables.
nix-repl> :b graphviz.override { xlibs = null; }
  • However, how do we make other packages refer to this overriden graphviz, instead of the original one?
    • There is no global, shared state to mutate – our only option is to re-evaluate everything in a modified context.
    • Note also, how not only the new derivations need to refer to the new graphviz, but they also must refer to the new versions of each other.
    • Hence the option of overriding everything manually would entail a prohibitive amount of work – basically amounting to repeating the whole repository with a new, parallel set of names.

Fixed point and beyond

  • Enter fix, a fix-point combinator, that:
    • takes a function
    • call it with its own returned value made available to it as argument
    fix = f: let result = f result; in result;
  • The effective call structure is f(f(f(....
  • If f always used the entirety of its argument, that would have been an infinite loop. 
    nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { x = "abc"; y = self.x + "123"; }
nix-repl> fix pkgs
{ x = "abc"; y = "abc123"; }

    Squinting a little, we can see how:

    1. the subexpression to the right of self is our repository declaration, where:
      • x and y correspond to our package names..
      • "abc", self.x + "123" correspond to our derivation expressions
    2. fix pkgs is the repository expression, yielding an attribute set mapping names to derivations
  • Note, how y is determined in terms of self – which isn’t quite pretty. Can we do without that, so we can squint less? Yes: 
    nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: with self; { x = "abc"; y = x + "123"; }
nix-repl> fix pkgs
{ x = "abc"; y = "abc123"; }
  • Notice how we didn’t even start with the overriding part, having merely established a correspondence between our toy example and Nixpkgs structure. So, what about overriding? Let’s note several details:
    1. We want to intervene at the level of repository expression, separately from the repository declaration, since pkgs has just the perfect structure, and we don’t want to mess with it.
    2. By the time we call fix, it’s too late to intervene – it seals the value.
    3. Adding up #1 and #2, we see that we need to produce an function that, given an overriding set, would transform pkgs, but without changing its type – the transformed value must be still something that we can pass to fix.
    4. Given our goals, we can postulate the following about this function:
      • it must take an overriding attribute set
      • it must take the object being overriden (a concrete value of pkgs)
      • it must return a function that would behave as pkgs does, which implies:
        • the returned function must take an attribute set – the value of self
        • the returned function must return an attribute set – with the overrides applied

    Here’s how this can be done:

    nix-repl> pkgs = self: with self; { x = "abc"; y = x + "123"; }
nix-repl> withOverride = overrides: f: self: f self // overrides
nix-repl> fix (withOverride { x = "def"; } pkgs)
{ x = "def"; y = "def123"; }
  • Now, we would like to use the package environment inside the override expression, wouldn’t we – for example to be able to override y? The only way to do that would be to change the override expression from being an attribute set to a function taking the package environment and returning the attribute set: 
    nix-repl> pkgs = self: with self; { x = "abc"; y = x + "123"; z = y + "456"; }
nix-repl> withOverride = overrides: f: self: f self // overrides self
nix-repl> fix (withOverride (self: with self; { y = x + "---"; }) pkgs)
{ x = "abc"; y = "abc---"; z = "abc---456"; }
  • The next step is to package the override method into the attribute set itself, in a way similar to how we did it in Nix Pill #14: 
    nix-repl> withOverride = overrides: f: self: f self // overrides self
nix-repl> virtual = let virt = f: fix f // { _override = overrides: virt (withOverride overrides f); }; in virt
nix-repl> pkgs = virtual (self: with self; { x = "abc"; y = x + "123"; z = y + "456"; })
nix-repl> pkgs
{ _override = «lambda»; x = "abc"; y = "abc123"; z = "abc123456"; }
nix-repl> pkgs._override (self: with self; { y = x + "---"; })
{ _override = «lambda»; x = "abc"; y = "abc---"; z = "abc---456"; }

Finally, we have produced a way to define an attribute set, with the following properties:

  • its definition is recursive – its elements can be defined in terms of each other
  • it ships with a method to perform a deep override of any element and, recursively, all the elements that refer to it

Overriding packages globally, the user-exposed way

Nixpkgs exposes the mechanism we examined above through the packageOverrides attribute in config / ~/.nixpkgs/config.nix:

{
  packageOverrides = pkgs: {
    graphviz = pkgs.graphviz.override { xlibs = null; };
  };
}

Overriding graphviz like this will make the change propagate to all packages that refer to it.

The ~/.nixpkgs/config.nix file

pkgs = import <nixpkgs> { config = import ./config.nix; }

Nix Pill #18: nix store paths

Origin: http://lethalman.blogspot.ru/2015/01/nix-pill-18-nix-store-paths.html

*** INCOMPLETE ***

About

Nix Pills: Compact Edition

Resources

Readme
Activity

Stars

24 stars

Watchers

6 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages