nixos-render-docs: init redirects system #353513

GetPsyched · 2024-11-03T22:17:53Z

An implementation of a redirects system to keep track of changes in the documentation such that old links don't break. Such a system will enable free movement of pages in the future without fear of breaking links; while also allowing us to catch old breakages retrospectively -- in a follow-up PR :D

PS: Commits will be squashed once the PR is marked for review.

pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/redirects.js

pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/redirects.py

pkgs/tools/nix/nixos-render-docs/src/tests/test_redirects.py

fricklerhandwerk · 2024-11-04T09:16:39Z

pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/redirects.py

+                path, anchor = location.split('#')
+                if path != redirection_target:
+                    continue
+                client_redirects[anchor] = f"{locations[0]}#{identifier}"


TODO for later: compute the relative path between redirection_target (maybe a better name: from?) and locations[0]. otherwise all of this will fall apart once we have files inside different directories

pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/redirects.py

pkgs/tools/nix/nixos-render-docs/src/tests/test_redirects.py

nixos-discourse · 2024-11-04T09:34:22Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/splitting-the-nixpkgs-and-nixos-manual-into-pages/55466/1

fricklerhandwerk

This looks good architecturally. Tests and build still fail though.

pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/redirects.py

pkgs/tools/nix/nixos-render-docs/src/tests/test_redirects.py

SuperSandro2000 · 2024-11-05T15:25:47Z

Do we have any options to not commit a 70k lines json? 😅

hsjobeki · 2024-11-05T17:02:07Z

Do we have any options to not commit a 70k lines json? 😅

It seems like the content is trivial too.
Could it be possible to compute the content as good as possible during the build if this file is needed.
I guess it is there for maintaining custom redirects. But it should be possible to only maintain what actually needs to be customized.

hsjobeki · 2024-11-05T17:07:08Z

@GetPsyched Thank you so much for doing this.

Once the pages are split i would like to integrate pagefind. This should be straight forward.
We should also export a sitemap xml i'll look into that so we also get better discoverability of our content by google.

GetPsyched · 2024-11-05T18:04:12Z

Do we have any options to not commit a 70k lines json? 😅

It seems like the content is trivial too. Could it be possible to compute the content as good as possible during the build if this file is needed. I guess it is there for maintaining custom redirects. But it should be possible to only maintain what actually needs to be customized.

I don't love that we're adding huge files when nixpkgs size is a growing concern; this is why a lot of effort went into making sure we have the most minimum diff.

We're compelled to storing all identifiers as its static data we cannot infer in the NRD's build process itself. One way to do it would be to check the base/parent commit, but that has its own issues; some that I can remember off the top of my head:

Non-trivial overhead to contributors who'll have to run external command(s) after each change.
Unsure how feasible is it to develop a CI check for this; and if there isn't one, contributors and reviewers are certainly going to forget to run such a command if there's no error to yell at them.

An alternative that was considered was storing each identifier as a file instead of a key in a JSON. We chose not to go that route as it would lead to having 20,000+ files as of 24.05.

GetPsyched · 2024-11-05T18:09:50Z

@GetPsyched Thank you so much for doing this.

Once the pages are split i would like to integrate pagefind. This should be straight forward. We should also export a sitemap xml i'll look into that so we also get better discoverability of our content by google.

Awesome! I've heard good things about pagefind and wanted to try it out myself at some point. IIRC it works off of the rendered HTML, right? This should fit quite well with the manuals.

I'll get to the XML bit once a few key things are checked off our todo; will ping you on the documentation Matrix room :D

pkgs/tools/nix/nixos-render-docs/README.md

hsjobeki · 2024-11-09T11:56:34Z

Do we have any options to not commit a 70k lines json? 😅

It seems like the content is trivial too. Could it be possible to compute the content as good as possible during the build if this file is needed. I guess it is there for maintaining custom redirects. But it should be possible to only maintain what actually needs to be customized.

I don't love that we're adding huge files when nixpkgs size is a growing concern; this is why a lot of effort went into making sure we have the most minimum diff.

We're compelled to storing all identifiers as its static data we cannot infer in the NRD's build process itself. One way to do it would be to check the base/parent commit, but that has its own issues; some that I can remember off the top of my head:

Non-trivial overhead to contributors who'll have to run external command(s) after each change.

Unsure how feasible is it to develop a CI check for this; and if there isn't one, contributors and reviewers are certainly going to forget to run such a command if there's no error to yell at them.

An alternative that was considered was storing each identifier as a file instead of a key in a JSON. We chose not to go that route as it would lead to having 20,000+ files as of 24.05.

I want to look into those details more closely. I am concerned about the size of the File. And huge diffs cluttering git. Please give me some time i ll do it then next days and provide alternatives If i find any

hsjobeki · 2024-11-12T16:32:11Z

If this gets too involved, the very least we could do is filtering out the library identifiers entirely and just not making any new guarantees for now. Actually, we could unblock this PR by merely filtering out all auto-identifiers and then adding redirect construction for them later.

That is what i would suggest.

I guess once the pages are split out the new automatic library and option docs will follow a certain schema.
The migration for those can be done alongside with splitting them.

Assuming that lib functions and options are stable. This will also yield stable redirects.

For breaking changes of the underlying references (lib, options) i would suggest that we introduce another kind of index for documentation that tracks lib functions that have been deprecated alongside with the reason and some other stuff. (Another PR, feature later)

The deprecation index for options should be possible to build from the options themselves somehow. But i would need to look into that details (option tree) a bit further. (Another PR, feature later)

pkgs/by-name/ni/nixos-render-docs/src/nixos_render_docs/redirects.py

fricklerhandwerk · 2024-11-13T12:13:47Z

@hsjobeki we're down to ~6400LOC for tracking redirects on hand-written content. Any objections to merging?

hsjobeki · 2024-11-13T13:25:26Z

@hsjobeki we're down to ~6400LOC for tracking redirects on hand-written content. Any objections to merging?

Seems as low as it may get for now. I am okay with merging as it is.

Could you add me as codeowner as well? So i get notified if anyone starts actively changing/adding redirects.

Also i feel like we are missing some documentation how to test a redirect after adding it. (nixos/nixpkgs manual)

ci/OWNERS

GetPsyched · 2024-11-13T16:06:51Z

Also i feel like we are missing some documentation how to test a redirect after adding it. (nixos/nixpkgs manual)

Can you elaborate on this? What would we write for such a document? (are we explaining how redirects work or how NRD implements them?)

fricklerhandwerk · 2024-11-13T17:10:40Z

Something like: cd doc; nix-build; firefox --new-tab file://$(pwd)/result/index.html#my-anchor and observe that it redirects

For server-side redirects (once we have them) you'd need to run netlify-cli like we do on nix.dev: https://github.com/NixOS/nix.dev/blob/master/CONTRIBUTING.md#local-preview. Or we could generate an nginx config and run a service; or a VM; it's Nix, we can do anything 🤣!

GetPsyched · 2024-11-15T08:29:58Z

NixOS and Nixpkgs manual CI is failing due to a lack of a rebase. I will do it once the changes have been reviewed.

This was unintentionally broken in 571c71e

Co-authored-by: Valentin Gagarin <[email protected]>

GetPsyched · 2024-11-15T19:14:21Z

I think we need to backport this PR since branch-off happened yesterday.
cc @fricklerhandwerk @hsjobeki

github-actions · 2024-11-15T19:45:19Z

Successfully created backport PR for release-24.11:

[Backport release-24.11] nixos-render-docs: init redirects system #356261

nixos-discourse · 2024-11-19T21:14:02Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/breaking-changes-announcement-for-unstable/17574/69

I find these `mkProvider` and `mkRequester` functions really nice. Also fixes doc generation related to NixOS/nixpkgs#353513

GetPsyched requested a review from fricklerhandwerk November 3, 2024 22:18

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation This PR adds or changes documentation labels Nov 3, 2024

fricklerhandwerk reviewed Nov 4, 2024

View reviewed changes