The original builtins.fetchGit
docs; see also comment in Nix issue #5128.
Quick notes beforehand:
-
The in-docs links (e.g., links to types such as "string", "attribute set", etc.) are just empty ones for now.
-
The inspiration for the "pseudo types" section come from the Erlang docs that is also a dynamically typed language, but has a "sub-language" for type specification for functions and each function doc starts with these, thus making the descriptions unambiguous.
improvement ideas:
Make "Types" section collapsible and add hover tooltip to expand type mentions in the descriptive portion of this reference.
Fix table 2.3-1's vertical header to the left so that it doesn't scroll out of screen
Link footnotes
Make typespec notes less prominent (italics doesn't seem to do much. CSS?)
...and questions:
- How to treat ancillary texts?
- Levels of detail?
builtins.fetchGit referenceToGitRepo -> storeResult
Fetch a Git repo.
referenceToGitRepo = URL | path | gitArgs
URL :: string = httpURL | httpsURL | ftpURL | fileURL
Supported code hosting services: GitHub, GitLab, SourceHut.
httpURL = string
Needs to conform to the http://
URI scheme (see RFC 9110, section 4.2).
httpsURL = string
Needs to conform to the https://
URI scheme (see RFC 9110, section 4.2).
ftpURL = string
Needs to conform to the ftp://
URI scheme (see RFC 1738, section 3.2).
webLikeURL :: string = httpURL | httpsURL | ftpURL
fileURL :: string = "file://"
+ fileURLPathPart
fileURLPathPart = string
fileURLPathPart is a shorthand for fileURL (i.e., it will be prefixed with "file://"
during evaluation) therefore both need to conform to the file://
URI scheme (see the path syntax of RFC 8089).
path = Nix path | fileURLPathPart
gitArgs :: attribute set =
{ url
:: (URL | path);
[ name
:: string ? "source"
];
[ ref
:: gitReference ? "HEAD"
];
[ rev
:: gitFullCommitHash ? <ref
dereferenced> ];
[ submodules
:: boolean ? false
];
[ shallow
:: boolean ? false
];
[ allRefs
:: boolean ? false
];
}
webLikeGitArgs :: attribute set =
(Erlang-y) gitArgs#{ url
:: webLikeURL; }
(Haskell-y) gitArgs { url
:: webLikeURL; }
pathLikeGitArgs :: attribute set =
(Erlang-y) gitArgs#{ url
:: (path | fileURL); }
(Haskell-y) gitArgs#{ url
:: (path | fileURL); }
webLike = webLikeURL | webLikeGitArgs
Argument that is a URL or has a URL member conforming to the http://
, https://
, and ftp://
URI schemes.
pathLike = path | fileURL | pathLikeGitArgs Argument that resolves or has a member that resolves to a file system path.
gitReference = string
Needs to be valid Git reference.
gitFullCommitHash = string
Has to be full SHA-1 (for now object name (40-byte hexadecimal string) that refers to an existing commit in the repo.
storeResult :: attribute set =
{ lastModified
:: ?;
lastModifiedDate
:: ?;
narHash
:: ?;
outPath
:: nixStorePath;
rev
:: gitFullCommitHash;
revCount
:: ?;
shortRev
:: ?;
submodules
:: boolean;
}
builtins.fetchGit
behaves differently when called with pathLike or webLike arguments.
These sections describe the behaviour of builtins.fetchGit
when called with webLike arguments:
-
1.1.1 webLikeURL type argument: string that conforms to the
http://
,https://
, andftp://
URI schemes. -
1.1.2 webLikeGitArgs type argument: same as gitArgs attribute set, except that the mandatory
url
attribute value is a webLikeURL
NOTE
The
file://
URI scheme is omitted on purpose, and is discussed in section 1.2 "Path-like" semantics.
builtins.fetchGit string
String format | Outcome | ||
---|---|---|---|
webLikeURL | httpURL | "http://..." |
The latest commit (or HEAD) of the repo's default branch (typically called main ormaster ) will be fetched. |
httpsURL | "https://..." |
||
ftpURL | "ftp://..." |
HTTPS examples with the supported code hosting sites:
builtins.fetchGit "https://github.com/NixOS/nix"
builtins.fetchGit "https://git.sr.ht/~rycee/configurations"
builtins.fetchGit "https://gitlab.com/rycee/home-manager"
NOTE
gitArgs
attributesrev
andref
will only be discussed in subsequent sections, but they also needed to be addressed here because of the significant role they play regarding the call results.
builtins.fetchGit attribute set
gitArgs attributes |
Outcome | Example argument | Example resolved to full webLikeGitArgs attribute set | ||
---|---|---|---|---|---|
url attribute (mandatory) |
rev attribute (optional) |
ref attribute (optional) |
|||
webLikeURL | omitted1.1.2-1 | omitted (or default value of HEAD used) |
Same asbuiltins.fetchGit webLikeURL(see Table 1.1.1-1 above) |
{ url = "https://github.com/nixos/nix"; } |
{ url = "https://github.com/NixOS/nix"; name = "source"; ref = "HEAD"; rev = "<SHA-1 commit hash of HEAD>" submodules = false; shallow = false; allRefs = false; } |
present | ignored1.1.2-2 | Fetch repo at rev commit |
{ url = "https://github.com/nixos/nix"; rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40"; } |
{ url = "https://github.com/nixos/nix"; name = "source"; ref = "" rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40"; submodules = false; shallow = false; allRefs = false; } |
|
omitted1.1.2-1 | present | Fetch repo at ref branch / tag |
{ url = "https://github.com/nixos/nix"; ref = "refs/tags/2.10.3"; } |
{ url = "https://github.com/nixos/nix"; name = "source"; ref = "refs/tags/2.10.3"; rev = "309c2e27542818b74219d6825e322b8965c7ad69"; submodules = false; shallow = false; allRefs = false; } |
[1.1.2-1]: See section 3.3 rev
[1.1.2-2]: See section 3.4 ref
Calls with pathLike arguments attempt to fetch a repo in a directory on a local or remote file system. The target repo may be a project under active development so their status and state may need to be determined before trying to copy the repo to the Nix store.
That is, characteristics that builtins.fetchGit
cares about.
The status of a Git repo is
-
dirty, if there are modified tracked files and/or staged changes. (Untracked content does not count.)
-
clean, if the output of
git diff-index HEAD
is empty. (If there are only untracked files ingit status
, the repo is clean.)
The state of a Git repo is the specific commit where the HEAD reference points to (directly or indirectly) at the moment when the repo is fetched.
Directly, if the repo is in a "detached HEAD" state, and indirectly when the commit is also the target of other references as shown on the figure below.
1.2.1.2-1. State of a Git repoLEGEND: orange label = branch, blue label = tag
1.2.2. Argument of type Nix path
, fileURL
, or fileURLPathPart
STATUS | De-reference process | |||
---|---|---|---|---|
dirty | clean | |||
STATE | on BRANCH |
Copy directory contents verbatim | Fetch repo at HEAD of BRANCH |
HEAD -> refs/heads/BRANCH -> <SHA-1 commit hash> |
at TAG |
Copy directory contents verbatim | Fetch repo at TAG |
HEAD -> refs/tags/TAG -> <SHA-1 commit hash> |
|
detached HEAD | Copy directory contents verbatim | Fetch repo at HEAD | HEAD -> <SHA-1 commit hash> |
In fact, the 3 "STATE" rows could easily be collapsed into one as Git branches and tags are only labels to a Git object and what matters to fetchGit
is the specific commit at the end of the de-reference process.
Example calls:
-
via Nix path:
builtins.fetchGit ~/clones/nix
-
via fileURL:
builtins.fetchGit "file:///home/nix_user/clones/nix"
-
via fileURLPathPart:
builtins.fetchGit "/home/nix_user/clones/nix"
This means one of the following:
-
via {
url
:: Nix path }builtins.fetchGit { url = ~/clones/nix; ... }
-
via {
url
:: fileURLPathPart }builtins.fetchGit { url = "/home/nix_user/clones/nix"; ... }
-
via {
url
:: fileURL }builtins.fetchGit { url = "file:///home/nix_user/clones/nix"; ... }
The following table takes advantage of the fact that state is simply determined by the current value of the HEAD reference:
NOTE
gitArgs
attributesrev
andref
will only be discussed in subsequent sections, but they also needed to be addressed here because of the significant role they play regarding the call results.
STATUS | gitArgs attributes |
Outcome | Example argument | ||
---|---|---|---|---|---|
url attribute (mandatory) |
rev attribute (optional) |
ref attribute (optional) |
|||
dirty | Nix path | fileURLPathPart | fileURL (See examples at the top of this section.) |
omitted1.2.3-1 | omitted (or default value of HEAD used) |
Copy directory contents verbatim |
{ url = "https://github.com/nixos/nix"; } |
clean | omitted1.2.3-1 | omitted (or default value of HEAD used) |
Fetch repo at HEAD | ||
ignored1.2.3-2 | present | ignored1.2.3-3 | Ignore changes (if any) and fetch repo at rev commit |
{ url = "https://github.com/nixos/nix"; rev = "be4654c344f0745d4c2eefb33a878bd1f23f6b40"; } |
|
ignored1.2.3-2 | omitted1.2.3-1 | present | Ignore changes (if any) and fetch repo at ref tag / branch |
{ url = "https://github.com/nixos/nix"; ref = "refs/tags/2.10.3"; } |
[1.2.3-1]: See section 3.3 rev
[1.2.3-2]: When ref
or rev
is present, the intention is probably to fetch a known past state from the repo's history, thus the most recent changes are not relevant (neither the status of the repo).
[1.2.3-3]: See section 3.4 ref
As a corollary, here are some tips:
-
If you need to fetch a local repo, calling
builtins.fetchGit
withref
(branch or tag) orrev
(commit hash) will make sure that a repo is fetched with a predictable content, ignoring any changes that may have been made since you last touched it. -
If you are packaging a project under active development and want to test changes without commiting, you'll probably want to call
builtins.fetchGit
with{ url = ...; }
or the specified in 1.2.2. Argument of typeNix path
,fileURL
, orfileURLPathPart
.
Reminder:
gitArgs :: attribute set =
{ url
:: (URL | path);
[ name
:: string ? "source"
];
[ ref
:: gitReference ? "HEAD"
];
[ rev
:: gitFullCommitHash ? <ref
dereferenced> ];
[ submodules
:: boolean ? false
];
[ shallow
:: boolean ? false
];
[ allRefs
:: boolean ? false
];
}
Description | This attribute is covered extensively in section 1. Behaviour (specifically, in sections 1.1.2 webLikeGitArgs type argument and 1.2.3 pathLikeGitArgs type argument). |
---|---|
Type | string |
Default value | none |
Description | The name part of the Nix store path where the Git repo's content will be copied to. |
---|---|
Type | string |
Default value | "source" |
Examples:
nix-repl> builtins.fetchGit { url = ./.; }
{ ...; outPath = "/nix/store/zwp1brk7ndhls3br4hk4h9xhpii17zs5-source"; ...; }
nix-repl> builtins.fetchGit { url = ./.; name = "miez"; }
{ ...; outPath = "/nix/store/zwp1brk7ndhls3br4hk4h9xhpii17zs5-miez"; ...; }
Description | The rev attribute is used to refer to a specific commit by the full SHA-1 Git object name (40-byte hexadecimal string) - or as it is more colloquially called, the commit hash. |
---|---|
Type | string |
Additional constraints |
40-byte hexadecimal SHA-1 string |
Default value | The dereferenced value of the Git reference held by the ref attribute. (See next section.) |
Sections 1.1.2 webLikeGitArgs type argument and 1.2.3 pathLikeGitArgs type argument) in 1. Behaviour describe the prevailing behaviour builtins.fetchgit
when the rev
attribute is used.
NOTE
Specifying the
rev
attribute will render theref
attribute irrelevant no matter if it is included in the input attribute set or not. See next section for more.
Description | The ref attribute accepts a Git reference that is present in the target repo.
|
---|---|
Type | string |
Additional constraints |
See Git reference syntax |
Default value | "HEAD" |
WARNING
By default, the
ref
value is prefixed withrefs/heads/
. After Nix 2.3.0, it will not be prefixed withrefs/heads/
ifref
starts withrefs/
.
3.3.1 ref
attribute ignored when the rev
attribute is provided
The rev
attribute (i.e., the commit hash) has higher specificity; a ref
reference will need to be resolved and its value may change with time, but a commit hash will always point to the same exact commit object and thus to the same state of the the repo during the lifetime of a Git repo. (TODO: right?)
TODO: Re-work original examples
TODO/NOTE: Stopping here for now to wait for the resolution of comment on Nix issue #5128
Here are some examples of how to use builtins.fetchGit
.
-
To fetch a private repository over SSH:
builtins.fetchGit { url = "[email protected]:my-secret/repository.git"; ref = "master"; rev = "adab8b916a45068c044658c4158d81878f9ed1c3"; }
-
To fetch an arbitrary reference:
builtins.fetchGit { url = "https://github.com/NixOS/nix.git"; ref = "refs/heads/0.5-release"; }
-
If the revision you're looking for is in the default branch of the git repository you don't strictly need to specify the branch name in the
ref
attribute.However, if the revision you're looking for is in a future branch for the non-default branch you will need to specify the the
ref
attribute as well.builtins.fetchGit { url = "https://github.com/nixos/nix.git"; rev = "841fcbd04755c7a2865c51c1e2d3b045976b7452"; ref = "1.11-maintenance"; }
Note
It is nice to always specify the branch which a revision belongs to. Without the branch being specified, the fetcher might fail if the default branch changes. Additionally, it can be confusing to try a commit from a non-default branch and see the fetch fail. If the branch is specified the fault is much more obvious.
-
If the revision you're looking for is in the default branch of the git repository you may omit the
ref
attribute.builtins.fetchGit { url = "https://github.com/nixos/nix.git"; rev = "841fcbd04755c7a2865c51c1e2d3b045976b7452"; }
-
To fetch a specific tag:
builtins.fetchGit { url = "https://github.com/nixos/nix.git"; ref = "refs/tags/1.9"; }
-
To fetch the latest version of a remote branch:
builtins.fetchGit { url = "ssh://[email protected]/nixos/nix.git"; ref = "master"; }
Note
Nix will refetch the branch in accordance with the option
tarball-ttl
.Note
This behavior is disabled in Pure evaluation mode.
TODO: move this to the end
[2.2-1]: See src/libfetchers/git.cc
#445
.
[2.2-2]: See src/libfetchers/git.cc
#fetch()
, line 556.
-
submodules
(optional) -
shallow
(optional) -
allRef
(optional)-
ref
(optional)
The git ref to look for the requested revision under. This is often a branch or tag name. Defaults toHEAD
. -
rev
(optional)
The Git revision to fetch.
Default value: if theref
attribute (see above) is specified, -
submodules
(optional)
A Boolean parameter that specifies whether submodules should be checked out. Defaults tofalse
. -
shallow
(optional)
A Boolean parameter that specifies whether fetching a shallow clone is allowed. Defaults tofalse
. NOTEgit clone --depth=1 <url>
creates a shallow cloneQUESTION How does this affect rev and ref? this is what i think
-
"no ref, no rev": ref=HEAD rev=resolve(HEAD) +++++++++clean repo: only diff is revCount The store hash stayed the same! Probably because .git is never copied - but then what is the point of this switch because then it will always be shallow. https://stackoverflow.com/questions/11497457/git-clone-without-git-directory QUESTION also, if .git is never copied, what is the point of revcount? NOTE should be called commitCount
nix-repl> builtins.fetchGit { url = ~/shed/my-project; } { lastModified = 1658846311; lastModifiedDate = "20220726143831"; narHash = "sha256-Yph6eCPxkG4TeoDAh/W6xaG+j5oFAui80c1FMYaGPTY="; outPath = "/nix/store/waffpfm7xrfyh1yj60v4phaf49ccyjd0-source"; rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f"; revCount = 5; shortRev = "5f45e9c"; submodules = false; } nix-repl> builtins.fetchGit { url = ~/shed/my-project; shallow = true; } { lastModified = 1658846311; lastModifiedDate = "20220726143831"; narHash = "sha256-Yph6eCPxkG4TeoDAh/W6xaG+j5oFAui80c1FMYaGPTY="; outPath = "/nix/store/waffpfm7xrfyh1yj60v4phaf49ccyjd0-source"; rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f"; revCount = 0; shortRev = "5f45e9c"; submodules = false; }
-
-
+++++++++dirty repo: completely identical outputs
nix-repl> builtins.fetchGit { url = ./.; }
warning: Git tree '/home/toraritte/clones/nix' is dirty
{ lastModified = 1658888196; lastModifiedDate = "20220727021636"; narHash = "sha256-bcPz3nYp1zK0HwwBqEybsxpDs5V7TPGUP3RE3Myd8zM="; outPath = "/nix/store/g0mrkf6vq0w6qzbkj03f4z3qhx18w50n-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; }
nix-repl> builtins.fetchGit { url = ./.; shallow = true; }
warning: Git tree '/home/toraritte/clones/nix' is dirty
{ lastModified = 1658888196; lastModifiedDate = "20220727021636"; narHash = "sha256-bcPz3nYp1zK0HwwBqEybsxpDs5V7TPGUP3RE3Myd8zM="; outPath = "/nix/store/g0mrkf6vq0w6qzbkj03f4z3qhx18w50n-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; }
+++++++++stopping this experiment here because adding rev and/or ref doesn't make a difference: only revCount will differ
-
allRefs
(optional)
Whether to fetch all refs of the repository. With this argument being true, it's possible to load arev
from anyref
(by default onlyrev
s from the specifiedref
are supported).NOTE allRefs also seems kind of pointless
nix-repl> builtins.fetchGit { url = ~/shed/my-project; allRefs = true; ref = "main";} { lastModified = 1658846059; lastModifiedDate = "20220726143419"; narHash = "sha256-FQUE8ek9uoyMy uGjQirYVc5B16X1Uq/k5e4LH+yv4S4="; outPath = "/nix/store/3ra7y3vsnbz707nq8r2d5p7k4irmiwrp-source"; rev = "c277976fce0b2b32b954a66d4345730b5b08f1db"; revCount = 3; shortRev = "c277976"; submodules = false; }
nix-repl> builtins.fetchGit { url = ~/shed/my-project; ref = "main";}
{ lastModified = 1658846059; lastModifiedDate = "20220726143419"; narHash = "sha256-FQUE8ek9uoyMy
uGjQirYVc5B16X1Uq/k5e4LH+yv4S4="; outPath = "/nix/store/3ra7y3vsnbz707nq8r2d5p7k4irmiwrp-source";
rev = "c277976fce0b2b32b954a66d4345730b5b08f1db"; revCount = 3; shortRev = "c277976"; submodules
= false; }
nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; } { lastModified = 1658489272; lastModifiedDate = "20220722112752"; narHash = "sha256-z0ov/NPT8egao DUVw4i5SuKcx6t7YZbL7lzdOBsP1sA="; outPath = "/nix/store/z13wfalqlfshjbkx5kwwgfm3350xnpdx-source"; rev = "280543933507839201547f831280faac614d0514"; revCount = 12454; shortRev = "2805439"; submod ules = false; }
nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; allRef = true; } error: unsupported Git input attribute 'allRef'
nix-repl> builtins.fetchGit { url = "https://github.com/nixos/nix"; allRefs = true; } { lastModified = 1658489272; lastModifiedDate = "20220722112752"; narHash = "sha256-z0ov/NPT8egao DUVw4i5SuKcx6t7YZbL7lzdOBsP1sA="; outPath = "/nix/store/z13wfalqlfshjbkx5kwwgfm3350xnpdx-source"; rev = "280543933507839201547f831280faac614d0514"; revCount = 12454; shortRev = "2805439"; submod ules = false; }
#####################
Only 2 uses of allRefs
in the entirety of Nixpkgs: TODO i don't think it matters if used or not; except if builtins.fetchGit
used "improperly" -> see issue below
0 [07:38:22] ag 'allRefs' . pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js 54: allRefs = true;
pkgs/development/tools/poetry2nix/poetry2nix/mk-poetry-dep.nix 179: allRefs = true;
#####################
NixOS/nix#5128 but the error makes sense as an SHA-1 hash is not a valid reference
nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; rev = "db1442a055 6c2b133627ffebf455a78a1ced64b9"; url = "https://github.com/tmcw/leftpad"; } fetching Git repository 'https://github.com/tmcw/leftpad'fatal: couldn't find remote ref refs/hea ds/db1442a0556c2b133627ffebf455a78a1ced64b9 error: program 'git' failed with exit code 128
nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; rev = "db1442a055 6c2b133627ffebf455a78a1ced64b9"; url = "https://github.com/tmcw/leftpad"; allRefs = true; } warning: could not update mtime for file '/home/toraritte/.cache/nix/gitv3/0240dfgnkwmgqs7sma8rns 8wlwxiv40b1lddl2sg2i0hnw7ym5c0/refs/heads/db1442a0556c2b133627ffebf455a78a1ced64b9': No such file or directory { lastModified = 1493781506; lastModifiedDate = "20170503031826"; narHash = "sha256-0DbZHwAdvEUiH o3brZyyxw0WdNQOsQwGZZz4tboN3v8="; outPath = "/nix/store/8frq54wwgi63wqgkc7p6yrcljlx4zwzh-source"; rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; revCount = 5; shortRev = "db1442a"; submodules = false; }
nix-repl>
nix-repl>
nix-repl> builtins.fetchGit { ref = "db1442a0556c2b133627ffebf455a78a1ced64b9"; url = "https://gi fetching Git repository 'https://github.com/tmcw/leftpad'fatal: couldn't find remote ref refs/hea ds/db1442a0556c2b133627ffebf455a78a1ced64b9 error: program 'git' failed with exit code 128
nix-repl> builtins.fetchGit { rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; url = "https://gi { lastModified = 1493781506; lastModifiedDate = "20170503031826"; narHash = "sha256-0DbZHwAdvEUiH o3brZyyxw0WdNQOsQwGZZz4tboN3v8="; outPath = "/nix/store/8frq54wwgi63wqgkc7p6yrcljlx4zwzh-source"; rev = "db1442a0556c2b133627ffebf455a78a1ced64b9"; revCount = 5; shortRev = "db1442a"; submodules = false; }