Skip to content

Latest commit

 

History

History
782 lines (621 loc) · 30.4 KB

fetchGit.md

File metadata and controls

782 lines (621 loc) · 30.4 KB

The original builtins.fetchGit docs; see also comment in Nix issue #5128.

Quick notes beforehand:

  • The in-docs links (e.g., links to types such as "string", "attribute set", etc.) are just empty ones for now.

  • The inspiration for the "pseudo types" section come from the Erlang docs that is also a dynamically typed language, but has a "sub-language" for type specification for functions and each function doc starts with these, thus making the descriptions unambiguous.

improvement ideas:

  • Make "Types" section collapsible and add hover tooltip to expand type mentions in the descriptive portion of this reference.

  • Fix table 2.3-1's vertical header to the left so that it doesn't scroll out of screen

  • Link footnotes

  • Make typespec notes less prominent (italics doesn't seem to do much. CSS?)

...and questions:

  • How to treat ancillary texts?
  • Levels of detail?

builtins.fetchGit referenceToGitRepo -> storeResult

Fetch a Git repo.

0. Pseudo Types

referenceToGitRepo = URL | path | gitArgs

URL :: string = httpURL | httpsURL | ftpURL | fileURL
  Supported code hosting services: GitHub, GitLab, SourceHut.

httpURL = string
  Needs to conform to the http:// URI scheme (see RFC 9110, section 4.2).

httpsURL = string
  Needs to conform to the https:// URI scheme (see RFC 9110, section 4.2).

ftpURL = string
  Needs to conform to the ftp:// URI scheme (see RFC 1738, section 3.2).

webLikeURL :: string = httpURL | httpsURL | ftpURL

fileURL :: string = "file://" + fileURLPathPart
fileURLPathPart = string
  fileURLPathPart is a shorthand for fileURL (i.e., it will be prefixed with "file://" during evaluation) therefore both need to conform to the file:// URI scheme (see the path syntax of RFC 8089).

path = Nix path | fileURLPathPart

gitArgs :: attribute set =
  { url :: (URL | path);
    [ name :: string ? "source" ];
    [ ref :: gitReference ? "HEAD" ];
    [ rev :: gitFullCommitHash ? <ref dereferenced> ];
    [ submodules :: boolean ? false ];
    [ shallow :: boolean ? false ];
    [ allRefs :: boolean ? false ];
  }

webLikeGitArgs :: attribute set =   (Erlang-y)  gitArgs#{ url :: webLikeURL; }   (Haskell-y) gitArgs { url :: webLikeURL; }

pathLikeGitArgs :: attribute set =   (Erlang-y)  gitArgs#{ url :: (path | fileURL); }   (Haskell-y) gitArgs#{ url :: (path | fileURL); }

webLike = webLikeURL | webLikeGitArgs   Argument that is a URL or has a URL member conforming to the http://, https://, and ftp:// URI schemes.

pathLike = path | fileURL | pathLikeGitArgs   Argument that resolves or has a member that resolves to a file system path.

gitReference = string
  Needs to be valid Git reference.

gitFullCommitHash = string
  Has to be full SHA-1 (for now object name (40-byte hexadecimal string) that refers to an existing commit in the repo.

storeResult :: attribute set =
  { lastModified :: ?;
    lastModifiedDate :: ?;
    narHash :: ?;
    outPath :: nixStorePath;
    rev :: gitFullCommitHash;
    revCount :: ?;
    shortRev :: ?;
    submodules :: boolean;
  }

1. Behaviour

builtins.fetchGit behaves differently when called with pathLike or webLike arguments.

1.1 "Web-like" semantics

These sections describe the behaviour of builtins.fetchGit when called with webLike arguments:

1.1.1 webLikeURL type argument

NOTE

The file:// URI scheme is omitted on purpose, and is discussed in section 1.2 "Path-like" semantics.

Table 1.1.1-1 builtins.fetchGit string
String format Outcome
webLikeURL httpURL "http://..." The latest commit (or HEAD)
of the repo's default branch
(typically called main or
master) will be fetched.
httpsURL "https://..."
ftpURL "ftp://..."

HTTPS examples with the supported code hosting sites:

builtins.fetchGit "https://github.com/NixOS/nix"
builtins.fetchGit "https://git.sr.ht/~rycee/configurations"
builtins.fetchGit "https://gitlab.com/rycee/home-manager"

1.1.2 webLikeGitArgs type argument

NOTE

gitArgs attributes rev and ref will only be discussed in subsequent sections, but they also needed to be addressed here because of the significant role they play regarding the call results.

Table 1.1.2-1 builtins.fetchGit attribute set
gitArgs
attributes
Outcome Example argument Example resolved to full webLikeGitArgs attribute set
url
attribute

(mandatory)
rev
attribute

   (optional)   
ref
attribute

   (optional)   
webLikeURL omitted1.1.2-1 omitted
(or default value of HEAD used)
Same as
builtins.fetchGit webLikeURL
(see Table 1.1.1-1 above)
{ url = "https://github.com/nixos/nix"; }
{
  url = "https://github.com/NixOS/nix";
  name = "source";
  ref = "HEAD";
  rev = "<SHA-1 commit hash of HEAD>"
  submodules = false;
  shallow = false;
  allRefs = false;
}
present ignored1.1.2-2 Fetch repo at rev commit
{ url = "https://github.com/nixos/nix";
  rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40";
}
{ url = "https://github.com/nixos/nix";
  name = "source";
  ref = ""
  rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40";
  submodules = false;
  shallow = false;
  allRefs = false;
}
omitted1.1.2-1 present Fetch repo at ref branch / tag
{ url = "https://github.com/nixos/nix";
  ref = "refs/tags/2.10.3";
}
{ url = "https://github.com/nixos/nix";
  name = "source";
  ref = "refs/tags/2.10.3";
  rev = "309c2e27542818b74219d6825e322b8965c7ad69";
  submodules = false;
  shallow = false;
  allRefs = false;
}

[1.1.2-1]: See section 3.3 rev

[1.1.2-2]: See section 3.4 ref

1.2 "Path-like" semantics

Calls with pathLike arguments attempt to fetch a repo in a directory on a local or remote file system. The target repo may be a project under active development so their status and state may need to be determined before trying to copy the repo to the Nix store.

1.2.1 Git repository characteristics

That is, characteristics that builtins.fetchGit cares about.

1.2.1.1 Status

The status of a Git repo is

  • dirty, if there are modified tracked files and/or staged changes. (Untracked content does not count.)

  • clean, if the output of git diff-index HEAD is empty. (If there are only untracked files in git status, the repo is clean.)

1.2.1.2 State

The state of a Git repo is the specific commit where the HEAD reference points to (directly or indirectly) at the moment when the repo is fetched.

Directly, if the repo is in a "detached HEAD" state, and indirectly when the commit is also the target of other references as shown on the figure below.

Visualized Git repo showing two branches and HEAD points to a commit that is tagged and is also the head of a branch.

1.2.1.2-1. State of a Git repo

LEGEND: orange label = branch, blue label = tag

1.2.2. Argument of type Nix path, fileURL, or fileURLPathPart

Table 1.2.2-1
STATUS De-reference process
dirty clean
STATE on BRANCH Copy directory contents verbatim Fetch repo at HEAD of BRANCH HEAD -> refs/heads/BRANCH -> <SHA-1 commit hash>
at TAG Copy directory contents verbatim Fetch repo at TAG HEAD -> refs/tags/TAG -> <SHA-1 commit hash>
detached HEAD Copy directory contents verbatim Fetch repo at HEAD HEAD -> <SHA-1 commit hash>

In fact, the 3 "STATE" rows could easily be collapsed into one as Git branches and tags are only labels to a Git object and what matters to fetchGit is the specific commit at the end of the de-reference process.

Example calls:

  • via Nix path:
    builtins.fetchGit ~/clones/nix

  • via fileURL:
    builtins.fetchGit "file:///home/nix_user/clones/nix"

  • via fileURLPathPart:
    builtins.fetchGit "/home/nix_user/clones/nix"

1.2.3 pathLikeGitArgs type argument

This means one of the following:

  • via { url :: Nix path } builtins.fetchGit { url = ~/clones/nix; ... }

  • via { url :: fileURLPathPart } builtins.fetchGit { url = "/home/nix_user/clones/nix"; ... }

  • via { url :: fileURL } builtins.fetchGit { url = "file:///home/nix_user/clones/nix"; ... }

The following table takes advantage of the fact that state is simply determined by the current value of the HEAD reference:

NOTE

gitArgs attributes rev and ref will only be discussed in subsequent sections, but they also needed to be addressed here because of the significant role they play regarding the call results.

Table 1.2.3-1.
    STATUS     gitArgs
attributes
        Outcome         Example argument
url
attribute

         (mandatory)         
rev
attribute

   (optional)   
ref
attribute

   (optional)   
dirty   Nix path
| fileURLPathPart
| fileURL

(See examples at the top of this section.)
omitted1.2.3-1 omitted
(or default value of HEAD used)
Copy directory contents verbatim
{ url = "https://github.com/nixos/nix"; }
clean omitted1.2.3-1 omitted
(or default value of HEAD used)
Fetch repo at HEAD
ignored1.2.3-2 present ignored1.2.3-3 Ignore changes (if any) and fetch repo at rev commit
{ url = "https://github.com/nixos/nix";
  rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40";
}
ignored1.2.3-2 omitted1.2.3-1 present Ignore changes (if any) and fetch repo at ref tag / branch
{ url = "https://github.com/nixos/nix";
  ref = "refs/tags/2.10.3";
}

[1.2.3-1]: See section 3.3 rev

[1.2.3-2]: When ref or rev is present, the intention is probably to fetch a known past state from the repo's history, thus the most recent changes are not relevant (neither the status of the repo).

[1.2.3-3]: See section 3.4 ref

As a corollary, here are some tips:

  • If you need to fetch a local repo, calling builtins.fetchGit with ref (branch or tag) or rev (commit hash) will make sure that a repo is fetched with a predictable content, ignoring any changes that may have been made since you last touched it.

  • If you are packaging a project under active development and want to test changes without commiting, you'll probably want to call builtins.fetchGit with { url = ...; } or the specified in 1.2.2. Argument of type Nix path, fileURL, or fileURLPathPart.

2. gitArgs attributes

Reminder:

gitArgs :: attribute set =
  { url :: (URL | path);
    [ name :: string ? "source" ];
    [ ref :: gitReference ? "HEAD" ];
    [ rev :: gitFullCommitHash ? <ref dereferenced> ];
    [ submodules :: boolean ? false ];
    [ shallow :: boolean ? false ];
    [ allRefs :: boolean ? false ];
  }

3.1 url (mandatory)

Description This attribute is covered extensively in section 1. Behaviour (specifically, in sections 1.1.2 webLikeGitArgs type argument and 1.2.3 pathLikeGitArgs type argument).
Type string
Default value none

3.2 name (optional)

Description The name part of the Nix store path where the Git repo's content will be copied to.
Type string
Default value "source"

Examples:

nix-repl> builtins.fetchGit { url = ./.; }
{ ...; outPath = "/nix/store/zwp1brk7ndhls3br4hk4h9xhpii17zs5-source"; ...; }

nix-repl> builtins.fetchGit { url = ./.; name = "miez"; }
{ ...; outPath = "/nix/store/zwp1brk7ndhls3br4hk4h9xhpii17zs5-miez"; ...; }

3.3 rev (optional)

Description The rev attribute is used to refer to a specific commit by the full SHA-1 Git object name (40-byte hexadecimal string) - or as it is more colloquially called, the commit hash.
Type string
Additional
constraints
40-byte hexadecimal SHA-1 string
Default value The dereferenced value of the Git reference held by the ref attribute. (See next section.)

Sections 1.1.2 webLikeGitArgs type argument and 1.2.3 pathLikeGitArgs type argument) in 1. Behaviour describe the prevailing behaviour builtins.fetchgit when the rev attribute is used.

NOTE

Specifying the rev attribute will render the ref attribute irrelevant no matter if it is included in the input attribute set or not. See next section for more.

3.4 ref (optional)

Description The ref attribute accepts a Git reference that is present in the target repo.
Type string
Additional
constraints
See Git reference syntax
Default value "HEAD"

WARNING

By default, the ref value is prefixed with refs/heads/. After Nix 2.3.0, it will not be prefixed with refs/heads/ if ref starts with refs/.

3.3.1 ref attribute ignored when the rev attribute is provided

The rev attribute (i.e., the commit hash) has higher specificity; a ref reference will need to be resolved and its value may change with time, but a commit hash will always point to the same exact commit object and thus to the same state of the the repo during the lifetime of a Git repo. (TODO: right?)

4. Examples

TODO: Re-work original examples


TODO/NOTE: Stopping here for now to wait for the resolution of comment on Nix issue #5128

Here are some examples of how to use builtins.fetchGit.

  • To fetch a private repository over SSH:

    builtins.fetchGit {
      url = "[email protected]:my-secret/repository.git";
      ref = "master";
      rev = "adab8b916a45068c044658c4158d81878f9ed1c3";
    }
  • To fetch an arbitrary reference:

    builtins.fetchGit {
      url = "https://github.com/NixOS/nix.git";
      ref = "refs/heads/0.5-release";
    }
  • If the revision you're looking for is in the default branch of the git repository you don't strictly need to specify the branch name in the ref attribute.

    However, if the revision you're looking for is in a future branch for the non-default branch you will need to specify the the ref attribute as well.

    builtins.fetchGit {
      url = "https://github.com/nixos/nix.git";
      rev = "841fcbd04755c7a2865c51c1e2d3b045976b7452";
      ref = "1.11-maintenance";
    }

    Note

    It is nice to always specify the branch which a revision belongs to. Without the branch being specified, the fetcher might fail if the default branch changes. Additionally, it can be confusing to try a commit from a non-default branch and see the fetch fail. If the branch is specified the fault is much more obvious.

  • If the revision you're looking for is in the default branch of the git repository you may omit the ref attribute.

    builtins.fetchGit {
      url = "https://github.com/nixos/nix.git";
      rev = "841fcbd04755c7a2865c51c1e2d3b045976b7452";
    }
  • To fetch a specific tag:

    builtins.fetchGit {
      url = "https://github.com/nixos/nix.git";
      ref = "refs/tags/1.9";
    }
  • To fetch the latest version of a remote branch:

    builtins.fetchGit {
      url = "ssh://[email protected]/nixos/nix.git";
      ref = "master";
    }

    Note

    Nix will refetch the branch in accordance with the option tarball-ttl.

    Note

    This behavior is disabled in Pure evaluation mode.

TODO: move this to the end [2.2-1]: See src/libfetchers/git.cc#445.

[2.2-2]: See src/libfetchers/git.cc#fetch(), line 556.

  • submodules (optional)

  • shallow (optional)

  • allRef (optional)

    • ref (optional)
      The git ref to look for the requested revision under. This is often a branch or tag name. Defaults to HEAD.

    • rev (optional)
      The Git revision to fetch.
      Default value: if the ref attribute (see above) is specified,

    • submodules (optional)
      A Boolean parameter that specifies whether submodules should be checked out. Defaults to false.

    • shallow (optional)
      A Boolean parameter that specifies whether fetching a shallow clone is allowed. Defaults to false. NOTE git clone --depth=1 <url> creates a shallow clone

      QUESTION How does this affect rev and ref? this is what i think

      • "no ref, no rev": ref=HEAD rev=resolve(HEAD) +++++++++clean repo: only diff is revCount The store hash stayed the same! Probably because .git is never copied - but then what is the point of this switch because then it will always be shallow. https://stackoverflow.com/questions/11497457/git-clone-without-git-directory QUESTION also, if .git is never copied, what is the point of revcount? NOTE should be called commitCount

        nix-repl> builtins.fetchGit { url = ~/shed/my-project; } { lastModified = 1658846311; lastModifiedDate = "20220726143831"; narHash = "sha256-Yph6eCPxkG4TeoDAh/W6xaG+j5oFAui80c1FMYaGPTY="; outPath = "/nix/store/waffpfm7xrfyh1yj60v4phaf49ccyjd0-source"; rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f"; revCount = 5; shortRev = "5f45e9c"; submodules = false; } nix-repl> builtins.fetchGit { url = ~/shed/my-project; shallow = true; } { lastModified = 1658846311; lastModifiedDate = "20220726143831"; narHash = "sha256-Yph6eCPxkG4TeoDAh/W6xaG+j5oFAui80c1FMYaGPTY="; outPath = "/nix/store/waffpfm7xrfyh1yj60v4phaf49ccyjd0-source"; rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f"; revCount = 0; shortRev = "5f45e9c"; submodules = false; }

+++++++++dirty repo: completely identical outputs

    nix-repl> builtins.fetchGit { url = ./.; }                 
    warning: Git tree '/home/toraritte/clones/nix' is dirty
    { lastModified = 1658888196; lastModifiedDate = "20220727021636"; narHash = "sha256-bcPz3nYp1zK0HwwBqEybsxpDs5V7TPGUP3RE3Myd8zM="; outPath = "/nix/store/g0mrkf6vq0w6qzbkj03f4z3qhx18w50n-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; }

    nix-repl> builtins.fetchGit { url = ./.; shallow = true; }
    warning: Git tree '/home/toraritte/clones/nix' is dirty
    { lastModified = 1658888196; lastModifiedDate = "20220727021636"; narHash = "sha256-bcPz3nYp1zK0HwwBqEybsxpDs5V7TPGUP3RE3Myd8zM="; outPath = "/nix/store/g0mrkf6vq0w6qzbkj03f4z3qhx18w50n-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; }

+++++++++stopping this experiment here because adding rev and/or ref doesn't make a difference: only revCount will differ

  • allRefs (optional)
    Whether to fetch all refs of the repository. With this argument being true, it's possible to load a rev from any ref (by default only revs from the specified ref are supported).

    NOTE allRefs also seems kind of pointless

nix-repl> builtins.fetchGit { url = ~/shed/my-project; allRefs = true; ref = "main";} { lastModified = 1658846059; lastModifiedDate = "20220726143419"; narHash = "sha256-FQUE8ek9uoyMy uGjQirYVc5B16X1Uq/k5e4LH+yv4S4="; outPath = "/nix/store/3ra7y3vsnbz707nq8r2d5p7k4irmiwrp-source"; rev = "c277976fce0b2b32b954a66d4345730b5b08f1db"; revCount = 3; shortRev = "c277976"; submodules = false; }

nix-repl> builtins.fetchGit { url = ~/shed/my-project; ref = "main";}
{ lastModified = 1658846059; lastModifiedDate = "20220726143419"; narHash = "sha256-FQUE8ek9uoyMy uGjQirYVc5B16X1Uq/k5e4LH+yv4S4="; outPath = "/nix/store/3ra7y3vsnbz707nq8r2d5p7k4irmiwrp-source"; rev = "c277976fce0b2b32b954a66d4345730b5b08f1db"; revCount = 3; shortRev = "c277976"; submodules = false; }

nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; } { lastModified = 1658489272; lastModifiedDate = "20220722112752"; narHash = "sha256-z0ov/NPT8egao DUVw4i5SuKcx6t7YZbL7lzdOBsP1sA="; outPath = "/nix/store/z13wfalqlfshjbkx5kwwgfm3350xnpdx-source"; rev = "280543933507839201547f831280faac614d0514"; revCount = 12454; shortRev = "2805439"; submod ules = false; }

nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; allRef = true; } error: unsupported Git input attribute 'allRef'

nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; allRefs = true; } { lastModified = 1658489272; lastModifiedDate = "20220722112752"; narHash = "sha256-z0ov/NPT8egao DUVw4i5SuKcx6t7YZbL7lzdOBsP1sA="; outPath = "/nix/store/z13wfalqlfshjbkx5kwwgfm3350xnpdx-source"; rev = "280543933507839201547f831280faac614d0514"; revCount = 12454; shortRev = "2805439"; submod ules = false; }

#####################

Only 2 uses of allRefs in the entirety of Nixpkgs: TODO i don't think it matters if used or not; except if builtins.fetchGit used "improperly" -> see issue below

0 [07:38:22] ag 'allRefs' . pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js 54: allRefs = true;

pkgs/development/tools/poetry2nix/poetry2nix/mk-poetry-dep.nix 179: allRefs = true;

#####################

NixOS/nix#5128 but the error makes sense as an SHA-1 hash is not a valid reference

nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; rev = "db1442a055 6c2b133627ffebf455a78a1ced64b9"; url = "https://github.com/tmcw/leftpad"; } fetching Git repository 'https://github.com/tmcw/leftpad'fatal: couldn't find remote ref refs/hea ds/db1442a0556c2b133627ffebf455a78a1ced64b9 error: program 'git' failed with exit code 128

nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; rev = "db1442a055 6c2b133627ffebf455a78a1ced64b9"; url = "https://github.com/tmcw/leftpad"; allRefs = true; } warning: could not update mtime for file '/home/toraritte/.cache/nix/gitv3/0240dfgnkwmgqs7sma8rns 8wlwxiv40b1lddl2sg2i0hnw7ym5c0/refs/heads/db1442a0556c2b133627ffebf455a78a1ced64b9': No such file or directory { lastModified = 1493781506; lastModifiedDate = "20170503031826"; narHash = "sha256-0DbZHwAdvEUiH o3brZyyxw0WdNQOsQwGZZz4tboN3v8="; outPath = "/nix/store/8frq54wwgi63wqgkc7p6yrcljlx4zwzh-source"; rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; revCount = 5; shortRev = "db1442a"; submodules = false; }

nix-repl>

nix-repl>

nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; url = "https://gi fetching Git repository 'https://github.com/tmcw/leftpad'fatal: couldn't find remote ref refs/hea ds/db1442a0556c2b133627ffebf455a78a1ced64b9 error: program 'git' failed with exit code 128

nix-repl> builtins.fetchGit { rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; url = "https://gi { lastModified = 1493781506; lastModifiedDate = "20170503031826"; narHash = "sha256-0DbZHwAdvEUiH o3brZyyxw0WdNQOsQwGZZz4tboN3v8="; outPath = "/nix/store/8frq54wwgi63wqgkc7p6yrcljlx4zwzh-source"; rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; revCount = 5; shortRev = "db1442a"; submodules = false; }