Allow rootless execution of repro #70

progandy · 2020-04-24T20:27:23Z

These changes allow to run repro without root privileges. Due to some issues, the resulting builds are not reproducible. This mode can be enabled with the -r flag

repro -r ...

Requirements

User Namespaces have to be allowed (sysctl kernel.unprivileged_userns_clone=1)
The user needs to have enough subuid/gid assigned in /etc/subuid and /etc/subgid. I used the following for both files:
```
progandy:100000:65536
```
become-root has to be installed.
nsjail did deny /proc/self/setgroups which caused issues with sudo/su. Maybe it is a kernel-bug that become-root works without that setting. I did not understand that section in the user_namespaces manpages completely.
It may be possible to get sudo (not su) working with setgroups disabled with the preserve_groups option.
nsjail is required instead of systemd-nspawn
fuse-overlayfs is required since overlayfs is not allowed for unprivileged namespaces in the upstream kernel.
unshare from util-linux. I use it to create a mount namespace for the container and as pid1 in the nsjail commands. The mount namespace might not be necessary and for pid1 it may be better to use dumb-init. I don't think it would be a good idea to directly run e.g. pacman as pid1, and I don't know about using bash.

Known Issues

In the build container, a downgrade of archlinux-keyring seems to fail signing keys, each operation times out, but the pacman command completes successfully.

(  2/126) downgrading archlinux-keyring    [###############################################] 100%
==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C...
==> ERROR: D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C could not be locally signed.

Installing the root container works fine.

(48/54) upgrading archlinux-keyring                [##############] 100%
==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C...
  -> Locally signing key DDB867B92AA789C165EEFA799B729B06A680C281...

fuse-overlayfs silently fails changing the modification time of symlinks.
(touch -h fails to set the date for symlinks containers/fuse-overlayfs#204)

│ │ -lrwxrwxrwx   0 root         (0) root         (0)        0 2020-04-07 11:35:48.000000 usr/bin/rnano -> nano
│ │ +lrwxrwxrwx   0 root         (0) root         (0)        0 2020-04-24 19:42:54.000000 usr/bin/rnano -> nano

Additional notes

I changed systemd-machine-id-setup to run inside the container, so rootless repro might run without systemd installed.

kpcyrd · 2020-04-25T02:05:49Z

Nice! Any chance you can submit cefd9a3 separately? It's a lot easier to land and would resolve #69.

progandy · 2020-04-25T09:49:11Z

Nice! Any chance you can submit cefd9a3 separately? It's a lot easier to land and would resolve #69.

Done: #73

FFY00 · 2020-04-25T13:45:06Z

become-root should get one release before we start relying on it. Perhaps you could ask the author.

progandy · 2020-04-25T15:40:44Z

Maybe I can get away without become-root. I don't want to add too many dependencies. That would mean repro would have to parse /etc/subuid or rely on the user to specify the allowed values.

The symlink issue will be fixed with containers/fuse-overlayfs#205.

The missing .MTREE might be a race condition in overlayfs or bash globs? It seems to work pretty well if I add sync .MTREE ; sleep 0.1 before the call to list_package_files in makepkg.

pacman-key --populate failed since the gpg-agent sockets were not properly cleaned up. and fuse-overlayfs could neither overwrite nor delete those inaccessible sockets. ~~I cannot reproduce it at the moment.~~ Now it is happening again.

Foxboron · 2020-04-26T15:51:07Z

I'm a bit skeptical of adding become_root as It's more dependencies which needs to be packaged. It's a bit unclear what functionality we'd need to replace though. Parsing /etc/subuid shouldn't be too hard in of itself.

progandy · 2020-04-26T16:05:31Z

nsjail always denies setgroups for the container. That means that su won't work and sudo only if it is configured with preserve_gropus. become-root did not have that problem, but I wonder if that is considered a bug du to this and this.

Foxboron · 2020-04-30T10:25:42Z

The fuse-overlayfs issue is fixed. Nice. So I'll try take a dig at how nsjail works and how we can try land this :)

progandy · 2020-04-30T10:40:58Z

Here are some more tests I did. I'm not sure if that is the best way, though. Maybe there should only be one nsjail/nspawn command that in turn runs a custom script instead of calling it for each command separately.

repro.in

Foxboron · 2020-04-30T10:51:06Z

As a sidenote, there is also https://github.com/containers/bubblewrap which tries some of the same things like nsjail.

Foxboron · 2020-04-30T10:52:14Z

Also, If you are able to rebase ontop of master it would be easier for me to test things.

progandy · 2020-04-30T10:58:38Z

It should be rebased already.

~~As far as I know, bubblewrap requires the SUID bit or has that changed?~~ I was trying to avoid requiring administrative as much as possible, but at minimum they are necessary to add the subuid/subgid entries with usermod.

Seems like bubblewarp can operate without SUID: https://github.com/containers/bubblewrap/blob/ff533b84d056f2c22633a84b34323dd085bd977a/bwrap.xml#L57

Foxboron · 2020-04-30T11:08:46Z

It's not rebased :)

progandy · 2020-04-30T11:51:26Z

Ah, rebased onto the wrong master branch. Now it should be ready.

Foxboron · 2020-04-30T12:12:36Z

Thanks for the work! I'll mark this as next on my todo and consider patching our fuse-overlayfs package

Foxboron · 2020-04-30T19:31:47Z

For testing purposes I have published fuse-overlayfs-1.0.0-2 which adds the symlink patch

Foxboron · 2020-04-30T20:15:41Z

Zomg! It works!

I managed to rebuild archlinux-keyring!

There are a few issues which I need to look at. /build in the container is created with uid 166535 instead of the correct one and missing /etc/resolv.conf so nameservers are not available. I have a few changes locally at the moment.

λ devtools-repro rootless» git diff
diff --git a/repro.in b/repro.in
index 759c4a9..d06b437 100755
--- a/repro.in
+++ b/repro.in
@@ -84,7 +84,7 @@ function require_userns_tools() {
 
 function mountoverlay() {
        if ((rootless_userns)); then
-               ~/Projekte/fuse-overlayfs/fuse-overlayfs "$@"
+               fuse-overlayfs "$@"
        else
                mount -t overlayfs overlayfs "$@"
        fi
@@ -285,6 +285,11 @@ function init_chroot(){
         mkdir -p "$BUILDDIRECTORY/root"
         tar xvf "$IMGDIRECTORY/$bootstrap_img" -C "$BUILDDIRECTORY/root" --strip-components=1 > /dev/null
 
+       # Custom rootless_userns configs here
+       if ((rootless_userns)); then
+           cat /etc/resolv.conf > "$BUILDDIRECTORY"/root/etc/resolv.conf
+       fi
+
         printf 'Server = %s\n' "$HOSTMIRROR" > "$BUILDDIRECTORY"/root/etc/pacman.d/mirrorlist
         printf '%s.UTF-8 UTF-8\n' en_US de_DE > "$BUILDDIRECTORY"/root/etc/locale.gen
         printf 'LANG=en_US.UTF-8\n' > "$BUILDDIRECTORY"/root/etc/locale.conf

Foxboron · 2020-04-30T20:17:16Z

repro.in

+}
+
+# Desc: Enter a user namespace with virtual privileges
+function become_rootless() {


I struggle to understand why we need to setup this as an replacement for sudo and su. And why we can't simply skip this and contain this logic to the exec_nsjail function

The reason I have this is that fuse-overlayfs has to run inside the user-namespace so it can map the subuids and I think some files inside the container filesystem are accessed from outside. With some refactoring becom_rootless can be removed I think, i.e. only call exec_nsjail twice. Once for the root setup and once for the build.

Foxboron · 2020-04-30T20:47:20Z

function build_package(){
## [snip]
    mkdir -p "./build"
    for pkgfile in "$BUILDDIRECTORY/$build"/pkgdest/*; do
        mv "$pkgfile" "./build/"
    done
    chown -R "$src_owner" "./build"
}

This logic doesn't work for rootless container. The persmission we end up with are from the container it seems

-rw-r--r-- 1 166535 166535 912011 Apr 30 22:45 archlinux-keyring-20200422-1-any.pkg.tar.zst

This is fixed by setting src_owner to "root" after doing the initial nsjail

progandy · 2020-04-30T21:02:03Z

This logic doesn't work for rootless container. The persmission we end up with are from the container it seems

-rw-r--r-- 1 166535 166535 912011 Apr 30 22:45 archlinux-keyring-20200422-1-any.pkg.tar.zst

While inside the user-namespace, it should be chowned to root, which will be mapped to the id of the user that created the namespace. I guess --keep-env in the first nsjail command causes problems. Either remove that and only pass necessary environment variables along or explicitly set $USER to root or something like that.

Foxboron · 2020-04-30T21:10:01Z

Current todo for this draft

Rebase against latest master still missing (also reindent from tab to 4 spaces)
Need to figure out what to do with /etc/resolv.conf

Any other things from your side which needs testing and/or some improvements? If the restructuring takes too much time or is complicated I don't think it's needed for this run.

Foxboron requested review from eli-schwartz, Foxboron, kpcyrd and anthraxx and removed request for eli-schwartz, Foxboron and kpcyrd April 24, 2020 20:34

progandy force-pushed the rootless branch from d78e845 to 9cb93f7 Compare April 25, 2020 09:47

progandy force-pushed the rootless branch from 9cb93f7 to 75823a1 Compare April 30, 2020 10:39

progandy commented Apr 30, 2020

View reviewed changes

repro.in Outdated Show resolved Hide resolved

progandy added 2 commits April 30, 2020 13:45

WIP: Implement rootless execution

e17ee44

some tests to only use nsjail

6934e99

progandy force-pushed the rootless branch from 75823a1 to 6934e99 Compare April 30, 2020 11:50

Foxboron reviewed Apr 30, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow rootless execution of repro #70

Allow rootless execution of repro #70

progandy commented Apr 24, 2020 •

edited

Loading

kpcyrd commented Apr 25, 2020

progandy commented Apr 25, 2020

FFY00 commented Apr 25, 2020

progandy commented Apr 25, 2020 •

edited

Loading

Foxboron commented Apr 26, 2020 •

edited

Loading

progandy commented Apr 26, 2020

Foxboron commented Apr 30, 2020

progandy commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron commented Apr 30, 2020

progandy commented Apr 30, 2020 •

edited

Loading

Foxboron commented Apr 30, 2020

progandy commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron Apr 30, 2020

progandy Apr 30, 2020

Foxboron commented Apr 30, 2020 •

edited

Loading

progandy commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Allow rootless execution of repro #70

Are you sure you want to change the base?

Allow rootless execution of repro #70

Conversation

progandy commented Apr 24, 2020 • edited Loading

Requirements

Known Issues

Additional notes

kpcyrd commented Apr 25, 2020

progandy commented Apr 25, 2020

FFY00 commented Apr 25, 2020

progandy commented Apr 25, 2020 • edited Loading

Foxboron commented Apr 26, 2020 • edited Loading

progandy commented Apr 26, 2020

Foxboron commented Apr 30, 2020

progandy commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron commented Apr 30, 2020

progandy commented Apr 30, 2020 • edited Loading

Foxboron commented Apr 30, 2020

progandy commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron commented Apr 30, 2020

Foxboron Apr 30, 2020

Choose a reason for hiding this comment

progandy Apr 30, 2020

Choose a reason for hiding this comment

Foxboron commented Apr 30, 2020 • edited Loading

progandy commented Apr 30, 2020

Foxboron commented Apr 30, 2020

progandy commented Apr 24, 2020 •

edited

Loading

progandy commented Apr 25, 2020 •

edited

Loading

Foxboron commented Apr 26, 2020 •

edited

Loading

progandy commented Apr 30, 2020 •

edited

Loading

Foxboron commented Apr 30, 2020 •

edited

Loading