Nixpkgs documentation: reword info about version testers

doc/build-helpers/testers.chapter.md

@@ -169,15 +169,23 @@ The build will fail if `shellcheck` finds any issues.
 
 Checks that the output from running a command contains the specified version string in it as a whole word.
 
-NOTE: In most cases, [`versionCheckHook`](#versioncheckhook) should be preferred, but this function is provided and documented here anyway. The motivation for adding either tests would be:
+Note: Given the current limitations of Nixpkgs's tooling and CI, `versionCheckHook` fits better than `testers.testVersion` in most of typical situations.
 
-- Catch dynamic linking errors and such and missing environment variables that should be added by wrapping.
-- Probable protection against accidentally building the wrong version, for example when using an "old" hash in a fixed-output derivation.
+Although being very simple and naive, this test is useful:
+
+- It assures the main program can run.
+- It brings awareness to dynamic linking errors, missing or malformed environment variables and other runtime errors.
+- It provides a minor protection against fixed-output hash errors that usually happen when forgetting to update hashes.
+
+By default, the command to be run will be inferred from the given `package` attribute: it will check `meta.mainProgram` first, falling back to `pname` or `name`.
 
-By default, the command to be run will be inferred from the given `package` attribute:
-it will check `meta.mainProgram` first, and fall back to `pname` or `name`.
 The default argument to the command is `--version`, and the version to be checked will be inferred from the given `package` attribute as well.
 
+Note: see also:
+
+- [`versionCheckHook`](#versioncheckhook), an independent hook with similar functionality.
+  - [A comparison between both](#versioncheckhook-comparison-testversion).
+
 :::{.example #ex-testversion-hello}
 
 # Check a program version using all the default values

doc/hooks/versionCheckHook.section.md

@@ -1,6 +1,12 @@
-# versionCheckHook {#versioncheckhook}
+1# versionCheckHook {#versioncheckhook}
 
-This hook adds a `versionCheckPhase` to the [`preInstallCheckHooks`](#ssec-installCheck-phase) that runs the main program of the derivation with a `--help` or `--version` argument, and checks that the `${version}` string is found in that output. You use it like this:
+This hook adds a `versionCheckPhase` to the [`preInstallCheckHooks`](#ssec-installCheck-phase) that runs the main program of the derivation with a `--help` or `--version` argument, and verifies if the `${version}` string is found in the output.
+
+It does so in a clean environment (using `env --ignore-environment --chdir=/`), and it checks for the `${version}` string in both the `stdout` and the `stderr` of the command. It will report to you in the build log the output it received and it will fail the build if it failed to find `${version}`.
+
+Here is a typical usage example:
+
+:::{.example #ex-versioncheckhook-phantom}
 
 ```nix
 {
@@ -16,20 +22,76 @@ stdenv.mkDerivation (finalAttrs: {
   nativeInstallCheckInputs = [
     versionCheckHook
   ];
+
   doInstallCheck = true;
 
   # ...
 })
 ```
 
-Note that for [`buildPythonPackage`](#buildpythonpackage-function) and [`buildPythonApplication`](#buildpythonapplication-function), `doInstallCheck` is enabled by default.
+:::
+
+Note that `doInstallCheck` is enabled by default for [`buildPythonPackage`](#buildpythonpackage-function) and [`buildPythonApplication`](#buildpythonapplication-function).
+
+## Attributes controlling versionCheckHook {#versioncheckhook-attributes-controlling}
+
+The following attributes control `versionCheckHook`:
+
+- `dontVersionCheck`: When set as `true`, the hook is not added to the [`preInstallCheckHooks`](#ssec-installCheck-phase).
+
+  Useful when one wants to load the function defined by the hook, so that it can be used for a different purpose.
+
+  Defaults to false.
+
+- `versionCheckProgram`: The full path to the program to be executed.
+
+   Defaults to `${placeholder "out"}/bin/${pname}`, roughly speaking.
+
+   The usage of `placeholder "out"` instead of `$out` here is unavoidable, since the hook does not expand environment variables.
+
+- `versionCheckProgramArg`: The argument to be passed to `versionCheckProgram`.
+
+  When not defined, the hook tries `--help` first and then `--version`.
+
+  Examples: `version`, `-V`, `-v`.
+
+- `preVersionCheck`: Hook that runs before the check is done. Defaults to an empty string.
+
+- `postVersionCheck`: Hook that runs after the check is done. Defaults to an empty string.
+
+Note: see also [`testers.testVersion`](#tester-testVersion).
+
+## Comparison between `versionCheckHook` and `testers.testVersion` {#versioncheckhook-testversion-comparison}
+
+The most notable difference between `versionCheckHook` and `testers.testVersion` is that `versionCheckHook` runs as a phase during the build time of the package, while `passthru.tests.versions` runs a complete derivation that depends on the package.
+
+Given the current limitations of Nixpkgs's tooling and CI, `versionCheckHook` fits better in most of typical situations.
+
+Below we provide tables comparing `versionCheckHook` and `testers.testVersion`.
+
+
+### General {#versioncheckhook-testversion-comparison-general}
+
+| Item                                      | `versionCheckHook`                                                                                                                                                      | `testers.testVersion` via `passthru.tests.version`                                                                                           |
+|-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
+| Customization                             | Besides the attributes described above, `versionCheckHook` provides no other methods for controlling it.                                                                | `tests.version` has access to the whole feature set of Nixpkgs, like multiple version test derivations and tools outside the package inputs. |
+| Execution time                            | `versionCheckHook` is executed automatically whenever the package is executed, e.g. `nix-build -A pkg`, since it is a phase executed during the package build time.     | `tests.version` is executed only when explicitly called, e.g. `nix-build -A pkg.tests.version`; it is executed after package build time.     |
+| Execution environment                     | `versionCheckHook` uses `env --ignore-environment --chdir=/` to run the executables in an environment as clean as possible, but there is no way to change its behavior. | `tests.version` executes in an environment identical to that of a consumer, independent from the package build environment.                  |
+| Rebuild after modification                | Modifying `versionCheckHook` triggers the package rebuild.                                                                                                              | `tests.version` does not rebuild the package, since it runs after package build time.                                                        |
+| Overhead during package build time        | Negligible. Although executed during the the package build time, `versionCheckHook` is lean and executes few commands.                                                  | Zero, since `tests.version` is a derivation that runs after package build time.                                                              |
+| Failure detection time                    | `versionCheckHook` runs during the package build time, therefore it does not deal with failures happening after that.                                                   | `tests.version` detects failures after package build time, like rewritings and relocations.                                                  |
+| Package breakage awareness                | Loud and clear as soon as `versionCheckHook` is reached during the package build time.                                                                                  | Requires specific commands to be noticed, e.g. `nix-build -A pkg.tests.version`.                                                             |
+
+### OfBorg {#versioncheckhook-testversion-comparison-general}
 
-It does so in a clean environment (using `env --ignore-environment`), and it checks for the `${version}` string in both the `stdout` and the `stderr` of the command. It will report to you in the build log the output it received and it will fail the build if it failed to find `${version}`.
+| Item                                      | `versionCheckHook`                                                                                                                                                      | `testers.testVersion` via `passthru.tests.version`                                                                                           |
+|-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
+| OfBorg CI                                 | `versionCheckHook` will be executed by OfBorg, since ofBorg automatically executes the packages affected by the pull request (PR).                                      | OfBorg supports automatic execution of `passthru.tests` for _packages directly affected_ by the PR.                                          |
+| OfBorg CI, transitive dependencies        | `versionCheckHook` will executed by OfBorg, since it builds transitive dependencies of a package affected by the PR. (????)                                             | OfBorg does not support automatic execution of `passthru.tests` for transitive dependencies,  requiring extra commands for this.             |
 
-The variables that this phase control are:
+### nixpkgs-review {#versioncheckhook-testversion-comparison-nixpkgs-review}
 
-- `dontVersionCheck`: Disable adding this hook to the [`preInstallCheckHooks`](#ssec-installCheck-phase). Useful if you do want to load the bash functions of the hook, but run them differently.
-- `versionCheckProgram`: The full path to the program that should print the `${version}` string. Defaults roughly to `${placeholder "out"}/bin/${pname}`. Using `$out` in the value of this variable won't work, as environment variables from this variable are not expanded by the hook. Hence using `placeholder` is unavoidable.
-- `versionCheckProgramArg`: The argument that needs to be passed to `versionCheckProgram`. If undefined the hook tries first `--help` and then `--version`. Examples: `version`, `-V`, `-v`.
-- `preVersionCheck`: A hook to run before the check is done.
-- `postVersionCheck`: A hook to run after the check is done.
+| Item                                      | `versionCheckHook`                                                                                                                                                      | `testers.testVersion` via `passthru.tests.version`                                                                                           |
+|-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
+| `nixpkgs-review`                          | `versionCheckHook` will be executed by `nixpkgs-review`.                                                                                                                | `nixpkgs-review` has no support for executing `passthru.tests` at all.                                                                       |
+| `nixpkgs-review`, transitive dependencies | `versionCheckHook` will be executed by `nixpkgs-review`, since the tool reaches transitive dependencies automatically.                                                  | `nixpkgs-review` has no support for executing `passthru.tests` at all,  even for transitive dependencies.                                    |

doc/stdenv/passthru.chapter.md

@@ -75,17 +75,18 @@ The Nixpkgs systems for continuous integration [Hydra](https://hydra.nixos.org/)
 #### Package tests {#var-passthru-tests-packages}
 []{#var-meta-tests-packages} <!-- legacy anchor -->
 
-Besides tests provided by upstream, that you run in the [`checkPhase`](#ssec-check-phase), you may want to define tests derivations in the `passthru.tests` attribute, which won't change the build. `passthru.tests` have several advantages over running tests during any of the [standard phases](#sec-stdenv-phases):
+Besides tests provided by upstream, usually executed during the [`checkPhase`](#ssec-check-phase), you may want to define test derivations in the `passthru.tests` attribute, which won't change the build. `passthru.tests` have several advantages over running tests during any of the [standard phases](#sec-stdenv-phases):
 
-- They access the package as consumers would, independently from the environment in which it was built
-- They can be run and debugged without rebuilding the package, which is useful if that takes a long time
-- They don't add overhead to each build, as opposed checks added to the [`installCheckPhase`](#ssec-installCheck-phase), such as [`versionCheckHook`](#versioncheckhook).
+- They are derivations themselves, therefore they have access to the whole feature set provided by Nix and Nixpkgs.
+  - For example, they can invoke extra tools, build reverse dependencies, fetch external files etc.
+- They access the package as consumers would, independently from the environment in which it was built.
+- They can be executed and debugged without rebuilding the package, which is useful when that rebuild takes a long time.
+- For content-addressed derivations, they test the executable __after__ rewriting instead of _before_, catching cases when the rewriting would break the executable.
+- They don't add overhead to each build, as opposed to checks added by the [`installCheckPhase`](#ssec-installCheck-phase), such as [`versionCheckHook`](#versioncheckhook).
 
-It is also possible to use `passthru.tests` to test the version with [`testVersion`](#tester-testVersion), but since that is pretty trivial and recommended thing to do, we recommend using [`versionCheckHook`](#versioncheckhook) for that, which has the following advantages over `passthru.tests`:
+Note: Especifically for version-testing, Nixpkgs provides [`testers.testVersion`](#tester-testVersion), traditionally used as `passthru.tests.version` attribute, as well as the `versionCheckHook` mentioned above.
 
-- If the `versionCheckPhase` (the phase defined by [`versionCheckHook`](#versioncheckhook)) fails, it triggers a failure which can't be ignored if you use the package, or if you find out about it in a [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) report.
-- Sometimes packages become silently broken - meaning they fail to launch but their build passes because they don't perform any tests in the `checkPhase`. If you use this tool infrequently, such a silent breakage may rot in your system / profile configuration, and you will not notice the failure until you will want to use this package. Testing such basic functionality ensures you have to deal with the failure when you update your system / profile.
-- When you open a PR, [ofborg](https://github.com/NixOS/ofborg)'s CI _will_ run `passthru.tests` of [packages that are directly changed by your PR (according to your commits' messages)](https://github.com/NixOS/ofborg?tab=readme-ov-file#automatic-building), but if you'd want to use the [`@ofborg build`](https://github.com/NixOS/ofborg?tab=readme-ov-file#build) command for dependent packages, you won't have to specify in addition the `.tests` attribute of the packages you want to build, and no body will be able to avoid these tests.
+Note: As shown in this [comparison](#versioncheckhook-comparison-testversion), `versionCheckHook` fits better than `testers.testVersion` in most of typical situations.
 
 <!-- NOTE(@fricklerhandwerk): one may argue whether that testing guide should rather be in the user's manual -->
 For more on how to write and run package tests for Nixpkgs, see the [testing section in the package contributor guide](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests).