Migrate packages and services into monorepo

[fricklerhandwerk]

In the past, Summer of Nix participants have packaged a large number of software projects in separate repositories. Now we want to migrate the individual packages and service modules into this monorepo or Nixpkgs.

The user-facing goal is present them as a coherent distribution (see #165).
The technical goal is to keep packages building against current Nixpkgs and have everything under consistent CI that can be maintained in one place.

Tasks

• Triage the complete list of repositories (or search ngi-nix repositories containing flake.nix)
• Delete unused forks
• Check if they are already in Nixpkgs or NixOS and maintained
• Open an issue on that repo if further clarification is needed
• Open pull requests referencing this tracking issue to migrate the contents of individual repositories into this monorepo
  • Open an issue referencing this tracking issue if a migration takes you more than two hours
• Add to the Migration project board the current issue/PR being worked on
• When done:
  • Delete the original Hydra job (if there's any)
  • Close the issue
  • Move the issue to the "Done" item list in the project board
  • Archive the original repo

Note

Some projects may have been dead for a long time. Please check first if they are still maintained, and ping me if in doubt. It's fine if you prioritise things you personally care about, as that makes an 80-20 result more likely and more pleasant to achieve.

Contact me when privileged access is required.

The desired end state is that there are only archived repositories apart from this monorepo and whatever isn't captured by the above activities.

[lorenzleutgeb]

In order to preserve the history of the repository that you are merging into ngipkgs, consider to not just copy the files, but doing the following history-preserving procedure (The example below is weblate. Use the repo name/slug, e.g. "foo" for ngi-nix/foo appropriately.):

1. Make sure you are on current main.
2. git remote add --fetch weblate [email protected]:ngi-nix/weblate.git (This will add a new remote called weblate, that will point to repository that you are migrating from. It will also fetch from that remote.)
3. git checkout -b weblate
4. git merge --allow-unrelated-histories weblate/main (assuming that the default branch of the repo to migrate from is called "main", maybe it uses "master" or similar)
5. The invocation of git merge will probably give you many conflicts. Don't just think about how to resolve them on the file level, but inspect the file structure of the repo you are migrating closely. There probably is one file that contains a package, which you will put into pkgs/weblate/default.nix or similar, and maybe one that contains a NixOS module, that will go to modules/weblate.nix. The same goes for tests. Don't think about this as usual conflict resolution, but you are now trying to make the repo fit ngipkgs. In case you have conflicts in flake.nix, you probably just want to restore the one from ngipkgs by executing git checkout main -- flake.nix. If the flake from the other repo contains interesting things, like the package definition itself, you can for example git show weblate/main:flake.nix > pkgs/weblate/default.nix to restore it at a better place. You can also of course move files around with git mv.
6. Don't forget to create entries in all-packages.nix and modules/all-modules.nix.
7. Format all files.
8. Bring the package into a state where nix flake show evaluates, and shows new derivations, but try to make as little changes as possible to get to that point.
9. Run git commit to conclude the merge.
10. Now you have a (maybe broken) package in ngipkgs. Fix it...

I advanced weblate up to step (9.) in parallel to writing this comment, see https://github.com/ngi-nix/ngipkgs/compare/weblate

Once you are done, you can of course git remote remove weblate.

The benefit of doing it this way, is that all commits from the repo being migrated are kept, including their hashes and dates. This way git blame will still work, references to PRs in commits are preserved, and generally the history of the repo with their original authors is preserved...

[fricklerhandwerk]

Another option is to use git subtree to get the entire history into the right subdirectory and then make a commit to wire up the package with the monorepo. That may be easier on the merge conflicts.

[lorenzleutgeb]

Another option is to use git subtree to get the entire history into the right subdirectory and then make a commit to wire up the package with the monorepo. That may be easier on the merge conflicts.

Yeah. I thought of this option. It just introduces one more commit, that's all.

[alejandrosame]

Some projects may have been dead for a long time. Please check first if they are still maintained, and ping me if in doubt. It's fine if you prioritise things you personally care about, as that makes an 80-20 result more likely and more pleasant to achieve.

I came to ask what is really the scope of this migration task. Are we increasing the visibility of all the repos inside the organisation or just the active ones?

For example, arpa2 packages seem to work with the current pinned dependencies (that is, nix build and nix flake check are successful). However, if the nixpkgs-unstable input is bumped to meet the current revision in the ngipkgs flake, the packages cannot be built anymore as it hasn't been touched for 2 years.

Should cases like arpa2 be dropped (after properly stating what's wrong with them)? Or are we aiming to bring these projects more inline with nixpkgs-unstable?

[fricklerhandwerk]

The goal is to keep packages building against current Nixpkgs. Let's prioritise low-hanging fruit: Projects that are well-maintained, popular, and quick to update. If something turns out to be a lot of work to fix, we'll figure out how to deal with that as we go. The only reason to drop a package would be if not maintained any more.

[lorenzleutgeb]

@fricklerhandwerk We've started merging migrated packages (#71) with more PRs approaching merge (e.g. #67). You mention that we should archive repos that we have migrated from. I suggest to insert a (generic) notice referring to https://github.com/ngi-nix/ngipkgs at the top of README.md in the repo that we migrated from:

This repository was archived. Its contents were migrated to <https://github.com/ngi-nix/ngipkgs>.

I'd like to start adding this line at the very top of README.md and archiving repos soon if there are no objections. At the moment I do not have permission to do so. So, for now I'll just collect a list of repos to archive.

[lorenzleutgeb]

I created the label migration and an issue template for migrations.

[augustebaum]

I would suggest creating a project board to track progress on this issue, as personally I have been (wrongly) using the "Summer of Nix" board for this purpose.