Migrate `hyperspace` #67

alejandrosame · 2023-10-08T19:23:23Z

This PR moves the Hyperspace monorepo into this monorepo.

To complete the migration, I made the following:

git subtree added the original repo.
checked that it would work with the same nixpkgs pinned in ngipkgs inputs.
extracted packages from original monorepo into by-name packages with the dream2nix V1 API (previous API is deprecated).
added installation checks for each individual package .

Closed because it was split into multiple PRs (rationale)[https://github.com//pull/67#issuecomment-1870770659]:

Tracking issue: #12.

Co-authored-by: Bill Ewanick <[email protected]> Co-authored-by: a-kenji <[email protected]> Co-authored-by: Matthias Meschede <[email protected]>

…8832436' git-subtree-dir: pkgs/hyperspace git-subtree-mainline: 55ee02e git-subtree-split: 26724a9

lorenzleutgeb · 2023-10-08T20:46:36Z

I have no experience with dream2nix, but I am asking myself whether package.nix and d2n-files/default.nix (per package) can be merged, so that we only have package.nix and even get rid of the d2n-files folder. That would be a lot leaner. I'd like to understand. Probably I should RTFM, but maybe you know and can tell me why the d2n-files/default.nix is required?

alejandrosame · 2023-10-09T13:26:54Z

@lorenzleutgeb I simply started from the examples in the current d2n repo. The reason it's there is because there are only examples of "drea2nix monorepos". So I already generalized it to contain only a package.

I can still try to merge them further, it might still be possible. I just made this PR after a bunch of refactors that left this code in a functional and OK state. I'll also ask for single package examples upstream.

Worst case, I could extract away the current boilerplate in a function and add it to the scope.

hyperspace: refactor to use dream2nix with by-name architecture

alejandrosame · 2023-10-14T17:57:51Z

@lorenzleutgeb with the help of @DavHau, this PR now makes it possible to add packages by-name and using dream2nix to evaluate them as modules.

In order to distinguish packages as modules from callPackage ones, the former need to be called module.nix inside it's named folder.

lorenzleutgeb

Awesome, looks much better!

The package names are missing version numbers. Is that intentional? Why?

nix flake show --json | jq -r '.packages."x86_64-linux"[].name' | sort

atomic-cli-0.34.5
atomic-server-0.34.5
autobase
corestore
flarum-1.8.0
gnunet-messenger-cli-0.1.1
hyperbeam
hyperblobs
hypercore
hyperswarm
lcrq
lcsync
libgnunetchat
librecast
nixos-system-server-23.11.20230919.8b5ab83
nixos-system-server-23.11.20230919.8b5ab83
nixos-system-test-flarum-23.11.20230919.8b5ab83
pretalx-2023.1.3
pretalx-2023.1.3
pretalx-frontend-2023.1.0
python3.10-euclid3-0.01
python3.10-kikit-1.3.0
python3.10-pcbnewTransition-0.3.4
python3.10-pretalx-downstream-1.1.5
python3.10-pretalx-media-ccc-de-1.1.1
python3.10-pretalx-pages-1.3.3
python3.10-pretalx-public-voting-1.3.0
python3.10-pretalx-venueless-1.3.0
python3.10-pybars3-0.9.7
python3.10-pymeta3-0.5.1
rosenpass-0.2.0
rosenpass-tools-0.2.0

I do understand that there are other packages that are also missing versions...

Also I saw that you have version = "v" + # ..., i.e., version prefixed with 'v' in some of the modules. Again, this is unusual. Any particular reason?

And lastly, there's no meta information at all, it seems. Could you please add that?

Co-authored-by: Alejandro Sanchez Medina <[email protected]>

augustebaum · 2023-10-24T14:24:43Z

@lorenzleutgeb We applied all of these suggestions, please have a look when you can.

lorenzleutgeb

Looking good! But I have one more request: Please reorder the contents of the modules. A logical order (for me), is the following:

Name and version.
Imports.
Sources.
Unpacking and Patching.
Dependencies (general definition, overrides, vendor hashes etc.)
Configuration of phases, in the order that they are executed.
Passthrus
Meta-Attributes.

I do understand that (4.) refers to phases, but these are the phases that are logically very tightly connected to definition of the source.

For this PR, I guess this boils down to moving name and version to the start of the module.

Again, this is my opinion, so if you don't agree you can reorder things in different ways. But a widespread convention is to put package name and version first, and meta-attributes last. I strongly recommend you do at least that.

Other than that, good to merge IMO!

This is because they time out after 1 hour even though they pass locally. Co-authored-by: Alejandro Sanchez Medina <[email protected]>

augustebaum · 2023-10-24T15:35:12Z

We ended up moving the name and version but not much else; as you said, with this dream2nix it's a bit harder to enforce the order of the callPackage convention.

lorenzleutgeb

Any particular reason to go for --debug? Seems excessive to me, --verbose should do.

lorenzleutgeb · 2023-10-25T13:49:24Z

.github/workflows/pull_request.yaml

@@ -15,4 +15,4 @@ jobs:
      - uses: DeterminateSystems/magic-nix-cache-action@main
        with:
          upstream-cache: https://ngi.cachix.org/
-      - run: nix flake check
+      - run: nix flake check --debug --print-build-logs


Suggested change

- run: nix flake check --debug --print-build-logs

- run: nix flake check --verbose --print-build-logs

This was added for testing as we cannot replicate the failure locally; we will probably revert this change when the issue is resolved.

lorenzleutgeb · 2023-10-25T13:49:35Z

.github/workflows/push.yaml

@@ -16,4 +16,4 @@ jobs:
        with:
          name: ngi
          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
-      - run: nix flake check
+      - run: nix flake check --debug --print-build-logs


Suggested change

- run: nix flake check --debug --print-build-logs

- run: nix flake check --verbose --print-build-logs

See #67 (comment).

OK. In such cases consider creating a new branch on top of the branch you are trying to merge, make the failure-diagnosis related changes in a new commit, and push to the new branch. That way there'll be no noise in the PR, and once you've resolved the issue, you can commit it to the PR branch. No big deal about the change in this instance, I am just trying to share a tip here.

Also note that this won't work for Actions that run on pull request only, like the one in .github/workflows/pull_request.yaml. That's why I am commenting here.

Leaving this unresolved, so that we don't forget to remove the debugging flags.

lorenzleutgeb · 2023-10-25T22:49:45Z

Regarding CI failures, I am slightly confused. Checks are green on e7be932, the last commit that actually has changes that are related to the PR. Checks fail on eb84483, the commit that "just" adds --debug --print-build-logs. The logs are very large (506MiB). The job is killed while it's compiling some Rust code. Probably because the logs are getting too big?! I don't think it's a timeout...

$ du -b log.txt
531484614       log.txt
$ head -n 6 log.txt
2023-10-25T13:44:51.0331804Z ##[group]Run nix flake check --debug --print-build-logs
2023-10-25T13:44:51.0332586Z nix flake check --debug --print-build-logs
2023-10-25T13:44:51.0379294Z shell: /usr/bin/bash -e {0}
2023-10-25T13:44:51.0379736Z env:
2023-10-25T13:44:51.0380281Z   MAGIC_NIX_CACHE_DAEMONDIR: /home/runner/work/_temp/magic-nix-cache-cMmTgQ
2023-10-25T13:44:51.0380970Z ##[endgroup]
$ tail -n 2 log.txt
2023-10-25T14:23:02.8720999Z rosenpass>    Compiling serde_derive v1.0.185
2023-10-25T14:23:20.1459141Z ##[error]The operation was canceled.

alejandrosame · 2023-12-28T02:41:32Z

After many tries to figure out what was wrong with CI failures, I decided to split the individual packages into separate PRs.

It was quite puzzling that #82 consistently worked with any combination of 5 packages but ended up failing when adding the 6th.

I took the opportunity and applied the naming convention proposed in #74. This means, using dream2.nix instead of using module.nix inside each by-name folder.

The list of PRs is:

alejandrosame · 2023-12-30T17:15:23Z

I'll go ahead and start merging the separate PRs, since this one was only stalled because CI never got green.

a-kenji and others added 15 commits August 31, 2022 14:11

first commit

f73a3ef

init: gitignore

5a2083d

init: flake.nix

3136c5d

Init hyperbeam flake

56b8e33

Co-authored-by: Bill Ewanick <[email protected]> Co-authored-by: a-kenji <[email protected]> Co-authored-by: Matthias Meschede <[email protected]>

Refactor

26724a9

Add 'pkgs/hyperspace/' from commit '26724a956c09fb89046d24271c170ab3f…

9d1b57d

…8832436' git-subtree-dir: pkgs/hyperspace git-subtree-mainline: 55ee02e git-subtree-split: 26724a9

hyperspace: fix build after syncing nixpkgs with parent flake lock

50295cc

hyperspace: migrate autobase package

3139f84

hyperspace: reformat nested flake.nix to comply with checks

823873a

hyperspace: refactor autobase package

76219dc

hyperspace: migrate hypercore package

cfd3833

hyperspace: migrate hyperbeam package

04b8a9c

hyperspace: migrate corestore package

faed418

hyperspace: migrate hyperblobs package

f5bfecd

hyperspace: migrate hyperswarm package

1b90321

alejandrosame requested review from lorenzleutgeb and fricklerhandwerk October 8, 2023 19:23

DavHau and others added 4 commits October 10, 2023 17:00

by-name: support dream2nix modules

fded826

autobase: migrate to using dream2nix by-name

693d9fa

Merge pull request #1 from DavHau/hyperspace

cdb7301

hyperspace: refactor to use dream2nix with by-name architecture

hyperspace: migrate packages to use dream2nix modules

cb104c5

lorenzleutgeb requested changes Oct 19, 2023

View reviewed changes

hyperspace: add meta and change package name

453a3d5

Co-authored-by: Alejandro Sanchez Medina <[email protected]>

lorenzleutgeb requested changes Oct 24, 2023

View reviewed changes

lorenzleutgeb mentioned this pull request Oct 24, 2023

Migrate packages and services into monorepo #12

Open

lorenzleutgeb added the migration label Oct 24, 2023

hyperspace: disable tests for hyperswarm

89ed722

This is because they time out after 1 hour even though they pass locally. Co-authored-by: Alejandro Sanchez Medina <[email protected]>

lorenzleutgeb changed the title ~~Migrate Hyperspace packages to monorepo~~ Migrate hyperspace Oct 24, 2023

lorenzleutgeb requested changes Oct 25, 2023

View reviewed changes

augustebaum assigned alejandrosame and augustebaum Oct 26, 2023

augustebaum marked this pull request as draft October 26, 2023 13:12

hyperspace: move name and version to the top

98f30c1

alejandrosame force-pushed the hyperspace branch from eb84483 to 98f30c1 Compare December 20, 2023 00:53

alejandrosame mentioned this pull request Dec 28, 2023

hyperspace-autobase: init at 1.0.0-alpha.9 #143

Merged

alejandrosame closed this Dec 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate `hyperspace` #67

Migrate `hyperspace` #67

alejandrosame commented Oct 8, 2023 •

edited by wamirez

Loading

lorenzleutgeb commented Oct 8, 2023

alejandrosame commented Oct 9, 2023

alejandrosame commented Oct 14, 2023

lorenzleutgeb left a comment •

edited

Loading

augustebaum commented Oct 24, 2023

lorenzleutgeb left a comment •

edited

Loading

augustebaum commented Oct 24, 2023

lorenzleutgeb left a comment

lorenzleutgeb Oct 25, 2023

augustebaum Oct 25, 2023

lorenzleutgeb Oct 25, 2023

augustebaum Oct 25, 2023

lorenzleutgeb Oct 25, 2023 •

edited

Loading

lorenzleutgeb commented Oct 25, 2023

alejandrosame commented Dec 28, 2023

alejandrosame commented Dec 30, 2023

	- run: nix flake check --debug --print-build-logs
	- run: nix flake check --verbose --print-build-logs

Migrate hyperspace #67

Migrate hyperspace #67

Conversation

alejandrosame commented Oct 8, 2023 • edited by wamirez Loading

lorenzleutgeb commented Oct 8, 2023

alejandrosame commented Oct 9, 2023

alejandrosame commented Oct 14, 2023

lorenzleutgeb left a comment • edited Loading

Choose a reason for hiding this comment

augustebaum commented Oct 24, 2023

lorenzleutgeb left a comment • edited Loading

Choose a reason for hiding this comment

augustebaum commented Oct 24, 2023

lorenzleutgeb left a comment

Choose a reason for hiding this comment

lorenzleutgeb Oct 25, 2023

Choose a reason for hiding this comment

augustebaum Oct 25, 2023

Choose a reason for hiding this comment

lorenzleutgeb Oct 25, 2023

Choose a reason for hiding this comment

augustebaum Oct 25, 2023

Choose a reason for hiding this comment

lorenzleutgeb Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

lorenzleutgeb commented Oct 25, 2023

alejandrosame commented Dec 28, 2023

alejandrosame commented Dec 30, 2023

Migrate `hyperspace` #67

Migrate `hyperspace` #67

alejandrosame commented Oct 8, 2023 •

edited by wamirez

Loading

lorenzleutgeb left a comment •

edited

Loading

lorenzleutgeb left a comment •

edited

Loading

lorenzleutgeb Oct 25, 2023 •

edited

Loading