Skip to content

Commit

Permalink
Rewrite documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@3XX0
3XX0 committed Apr 24, 2019
1 parent 5bb2911 commit 2c410f2
Show file tree
Hide file tree
Showing 16 changed files with 718 additions and 238 deletions.
272 changes: 35 additions & 237 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,256 +1,54 @@
# ENROOT

A set of scripts and utilities to run container images as unprivileged "chroot" (or what some people refer to as HPC containers).
A simple, yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.

Example:
```bash
enroot import docker://alpine
enroot create alpine.sqsh
enroot start alpine
```

## Prerequisites

Kernel settings:
```bash
# Make sure your kernel is fairly recent (~ 3.10) and supports what's required:
grep -E '(CONFIG_NAMESPACES|CONFIG_USER_NS|CONFIG_OVERLAY_FS|CONFIG_SECCOMP_FILTER)=' /boot/config-$(uname -r)
# For running old containers (e.g. Centos 6) on new kernels (4.8+)
grep 'CONFIG_X86_VSYSCALL_EMULATION'/boot/config-$(uname -r) && grep 'vsyscall=emulate' /proc/cmdline

# Configure namespace limits appropriately if necessary
sudo tee -a /etc/sysctl.d/10-namespace.conf <<< "user.max_user_namespaces = 65536"
sudo tee -a /etc/sysctl.d/10-namespace.conf <<< "user.max_mnt_namespaces = 65536"

# Debian distributions
sudo tee -a /etc/sysctl.d/10-namespace.conf <<< "kernel.unprivileged_userns_clone = 1"

# RHEL distributions
sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

sudo reboot
```

Dependencies:
```bash
# Debian distributions (build, required, optional)
sudo apt install -y gcc make libcap2-bin libbsd-dev
sudo apt install -y curl tar pigz jq squashfs-tools parallel
sudo apt install -y pv nvidia-container-cli

# RHEL distributions (build, required, optional)
sudo yum install -y epel-release
sudo yum install -y gcc make libcap libbsd-devel
sudo yum install -y curl tar pigz jq squashfs-tools parallel
sudo yum install -y pv nvidia-container-cli
````

## Installation

```bash
git submodule update --init
sudo make install
# In order to allow unprivileged users to import images
sudo make setcap
```

Environment settings:

| Environment | Default | Description |
| ------ | ------ | ------ |
| `ENROOT_LIBEXEC_PATH` | `/usr/local/libexec/enroot` | Path to sources and utilities |
| `ENROOT_SYSCONF_PATH` | `/usr/local/etc/enroot` | Path to system configuration files |
| `ENROOT_CONFIG_PATH` | `${XDG_CONFIG_HOME}/enroot` | Path to user configuration files |
| `ENROOT_CACHE_PATH` | `${XDG_CACHE_HOME}/enroot` | Path to user image/credentials cache |
| `ENROOT_DATA_PATH` | `${XDG_DATA_HOME}/enroot` | Path to user container storage |
| `ENROOT_RUNTIME_PATH` | `${XDG_RUNTIME_DIR}/enroot` | Path to the runtime working directory |

## Usage
```
Usage: enroot COMMAND [ARG...]

Commands:
version
import [--output|-o IMAGE] URI
export [--output|-o IMAGE] NAME
create [--name|-n NAME] IMAGE
list [--fancy|-f]
remove [--force|-f] NAME...
start [--root|-r] [--rw|-w] [--conf|-c CONFIG] NAME [COMMAND] [ARG...]
bundle [--all|-a] [--output|-o BUNDLE] [--checksum|-c] [--target|-t TARGET] [--desc|-d TEXT] IMAGE
```
## Commands
### version
Show the version of Enroot.
### import
Import and convert a Docker container image to an [Enroot image](#image-format) where the URI is of the form
`docker://[<user>@][<registry>#]<image>[:<tag>]`. Digests will be cached under `${ENROOT_CACHE_PATH}`.
Credentials can be configured by writing to the file `${ENROOT_CONFIG_PATH}/.credentials` following the netrc file format. For example:
```
# NVIDIA GPU Cloud
machine authn.nvidia.com login $oauthtoken password <TOKEN>
# Docker Hub
machine auth.docker.io login <LOGIN> password <PASSWORD>
```
Enroot can be thought of as an enhanced unprivileged `chroot(1)`. It uses the same underlying technologies as containers but removes much of the isolation they inherently provide while preserving filesystem separation.

Environment settings:
This approach is generally preferred in high-performance environments or virtualized environments where portability and reproducibility is important, but extra isolation is not warranted.

| Environment | Default | Description |
| ------ | ------ | ------ |
| `ENROOT_GZIP_PROG` | `pigz` _or_ `gzip` | Gzip program used to uncompress digest layers |
| `ENROOT_SQUASH_OPTS` | `-comp lzo -noD` | Options passed to `mksquashfs` to produce the image |
Enroot is also similar to other tools like `proot(1)` or `fakeroot(1)` but instead relies on more recent features from the Linux kernel (i.e. user and mount namespaces), and provides facilities to import well known container image formats (e.g. [Docker](https://www.docker.com/)).

### export
Export a container root filesystem found under `${ENROOT_DATA_PATH}` to a container image.
The resulting artifact can then be unpacked using the [create](#create) command.
Usage example:

Environment settings:
| Environment | Default | Description |
| ------ | ------ | ------ |
| `ENROOT_SQUASH_OPTS` | `-comp lzo -noD` | Options passed to `mksquashfs` to produce the image |
### create
Take a container image and unpack its root filesystem under `${ENROOT_DATA_PATH}` with the given name (optionally).
The resulting artifact can then be started using the [start](#start) command.
### list
List all the containers along with their size on disk (optionally).
### remove
Remove a container, deleting its root filesystem from disk.
### start
Start a previously [created](#create) container by executing its command script (or entrypoint), refer to [Image format (/etc/rc)](#image-format).
By default the root filesystem of the container is made read-only unless the `--rw` option has been provided.
The `--root` option can also be provided in order to remap your current user to be root inside the container.
Additionally, a configuration script can be provided with `--conf` to perform specific actions before the container starts.
This script is a standard bash script called before any configuration happens with the command and arguments passed as input parameters.
One or more of the following functions can be defined:
| Function | Description |
| ------ | ------ |
| `environ()` | Outputs [environment configuration](#environment-configuration-files) |
| `mounts()` | Outputs [mount configuration](#mount-configuration-files) |
| `hooks()` | A specific instance of [pre-start hook scripts](#pre-start-hook-scripts) |
Here is an example of such configuration:
```bash
environ() {
# Keep all the environment from the host
env
}
mounts() {
# Mount the X11 unix-domain socket
echo "/tmp/.X11-unix /tmp/.X11-unix none x-create=dir,bind"
# Mount the current working directory to /mnt
echo "${PWD} /mnt none bind"
}
hooks() {
# Set the DISPLAY environment variable if not set
[ -z "${DISPLAY-}" ] && echo "DISPLAY=:0.0" >> ${ENROOT_ENVIRON}
# Record the date when the container was last started
date > ${ENROOT_ROOTFS}/last_started
}
```

Environment settings:

| Environment | Default | Description |
| ------ | ------ | ------ |
| `ENROOT_LOGIN_SHELL` | `/bin/sh` | Login shell used to run the container initialization (i.e. `/init`)|
| `ENROOT_ROOTFS_RW` | | Equivalent to `--rw` if set |
| `ENROOT_REMAP_ROOT` | | Equivalent to `--root` if set |

### bundle
Create a self-extracting bundle from a container image which can be used to start a container with no external dependencies (on most Linux distributions).
The resulting bundle takes the same arguments as the [start](#start) command with the addition of `--info` which displays the bundle information, and
`--keep` which keeps the container filesystem extracted to the target directory after exiting. If `--keep` was not provided, `${TMPDIR}` is used for extraction.

By default, only system-wide configuration is copied to the bundle unless `--all` is specified, in which case user-specified configuration is copied as well.
The target directory used to keep the container filesystem can be defined using the `--target` option and defaults to `${PWD}/<bundle>`.
Additionally, a checksum can be generated and a description provided with `--checksum` and `--desc` respectively.

Example:
```bash
enroot import docker://alpine
enroot bundle -t '${HOME}/.local/share/enroot/foobar' alpine.sqsh
./alpine.run --keep --rw cp /etc/os-release /release
enroot start foobar cat /release
```sh
# Import and start an Ubuntu image from DockerHub
$ enroot import docker://ubuntu
$ enroot create ubuntu.sqsh
$ enroot start ubuntu
```

Environment settings:

| Environment | Default | Description |
| ------ | ------ | ------ |
| `ENROOT_BUNDLE_ALL` | | Equivalent to `--all` if set |
| `ENROOT_BUNDLE_SUM` | | Equivalent to `--checksum` if set |

## Image format
Enroot images are standard squashfs images with the following configuration files influencing runtime behaviors.

| File | Description |
| ------ | ------ |
| `/etc/rc` | Command script of the container (entrypoint) |
| `/etc/fstab` | Mount configuration of the container |
| `/etc/environment` | Environment of the container |
## Key Concepts

These files follow the same format as the standard Linux/Unix ones (see _fstab(5)_, _rc(8)_, _pam_env(8)_) with the exceptions listed below.

`/etc/rc`:
- The command and arguments of the [start](#start) command are passed as input parameters.


`/etc/fstab`:
- Adds two additional mount options, `x-create=dir` or `x-create=file` to create an empty directory or file before performing the mount.
- The target mountpoint is relative to the container rootfs.
- References to environment variables from the host of the form `${ENVVAR}` will be substituted

```
# Example mounting your home directory from the host
${HOME} ${HOME} none x-create=dir,bind
```
* Adheres to the [KISS principle](https://en.wikipedia.org/wiki/KISS_principle) and [Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy)
* Standalone (no daemon)
* Fully unprivileged and multi-user capable (no setuid binary, cgroup inheritance, per-user configuration/container store...)
* Easy to use (simple image format, scriptable, root remapping...)
* Little to no isolation (no performance overhead, simplifies HPC deployements)
* Entirely composable and extensible (system-wide and user-specific configurations)
* Fast Docker image import (3x to 5x speedup on large images)
* Built-in GPU support with [libnvidia-container](https://github.com/nvidia/libnvidia-container)
* Facilitate collaboration and development workflows (bundles, in-memory containers...)

`/etc/environment`:
- References to environment variables from the host of the form `${ENVVAR}` will be substituted
## Documentation

```bash
# Example preserving the DISPLAY environment variable from the host
DISPLAY=${DISPLAY}
```
1. [Requirements](doc/requirements.md)
1. [Installation](doc/installation.md)
1. [Image format](doc/image-format.md)
1. [Configuration](doc/configuration.md)
1. [Standard Hooks](doc/standard-hooks.md)
1. [Usage](doc/usage.md)

## Configuration

Common configurations can be applied to all containers by leveraging the following directories under `${ENROOT_SYSCONF_PATH}` (system-wide) and/or `${ENROOT_CONFIG_PATH}` (user-specific).
## Copyright and License

| Directory | Description |
| ------ | ------ |
| `environ.d` | Environment configuration files |
| `mounts.d` | Mount configuration files |
| `hooks.d` | Pre-start hook scripts |
This project is released under the [BSD 3-clause license](https://github.com/NVIDIA/enroot/blob/master/LICENSE).

### Environment configuration files
Environment files have the `.env` extension and follow the same format as described in [Image format (/etc/environment)](#image-format)
## Issues and Contributing

### Mount configuration files
Mount files have the `.fstab` extension and follow the same format as described in [Image format (/etc/fstab)](#image-format)
* Please let us know by [filing a new issue](https://github.com/NVIDIA/enroot/issues/new)
* You can contribute by opening a [pull request](https://help.github.com/articles/using-pull-requests/)

### Pre-start hook scripts
Pre-start hooks are standard bash scripts with the `.sh` extension.
They run with full capabilities before the container has switched to its final root.
Scripts are started with the host environment as well as the following environment variables:
## Reporting Security Issues

| Environment | Description |
| ------ | ------ |
| `ENROOT_PID` | PID of the container |
| `ENROOT_ROOTFS` | Path to the container rootfs |
| `ENROOT_ENVIRON` | Path to the container environment file to be read at startup |
When reporting a security issue, do not create an issue or file a pull request.
Instead, disclose the issue responsibly by sending an email to `psirt<at>nvidia.com`.
2 changes: 1 addition & 1 deletion conf/enroot.conf.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
# Remap the current user to root inside containers by default.
#ENROOT_REMAP_ROOT no

# Maximum time in seconds to wait for connections estabishment.
# Maximum time in seconds to wait for connections establishment.
#ENROOT_CONNECT_TIMEOUT 30

# Use HTTP for outgoing requests instead of HTTPS (UNSECURE!).
Expand Down
62 changes: 62 additions & 0 deletions doc/cmd/bundle.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Usage
```
Usage: enroot bundle [options] [--] IMAGE
Create a self-extracting bundle from a container image.
Options:
-a, --all Include user configuration files in the bundle
-c, --checksum Generate an embedded checksum
-d, --desc TEXT Provide a description of the bundle
-o, --output BUNDLE Name of the output bundle file (defaults to "IMAGE.run")
-t, --target DIR Target directory used by --keep (defaults to "$PWD/BUNDLE")
```
```
Usage: enroot-check_1.0.0.run [options] [--] [COMMAND] [ARG...]
Options:
-i, --info Display the information about this bundle
-k, --keep Keep the bundle extracted in the target directory
-q, --quiet Supress the progress bar output
-v, --verify Verify that the host configuration is compatible with the bundle
-x, --extract Extract the bundle in the target directory and exit (implies --keep)
-c, --conf CONFIG Specify a configuration script to run before the container starts
-e, --env KEY[=VAL] Export an environment variable inside the container
-m, --mount FSTAB Perform a mount from the host inside the container (colon-separated)
-r, --root Ask to be remapped to root inside the container
-w, --rw Make the container root filesystem writable
```

# Description

Create a self-extracting bundle from a container image which can be used to start a container on most Linux distributions with no external dependencies.
The resulting bundle takes the same arguments as the [start](start.md) command.

The target directory used to keep the container filesystem can be defined using the `--target` option and defaults to `$PWD/<bundle>`.
By default when generating a bundle, only system-wide configuration is copied to the bundle unless `--all` is specified, in which case user-specified configuration is copied as well.

Before executing a bundle, `--verify` can be used to check whether or not the host meets the necessary requirements.

If `--keep` is not provided at launch, `$ENROOT_TEMP_PATH` will be used for extraction.

# Configuration

| Environment | Default | Description |
| ------ | ------ | ------ |
| `ENROOT_BUNDLE_ALL` | `no` | Include user-specific configuration inside bundles (same as `--all`) |
| `ENROOT_BUNDLE_CHECKSUM` | `no` | Generate an embedded checksum inside bundles (same as `--checksum`) |

# Example

```sh
# Import Ubuntu from DockerHub and generate a bundle from it
$ enroot import docker://ubuntu
$ enroot bundle --target '${HOME}/.local/share/enroot/hello-world' ubuntu.sqsh

# Execute the bundle by writing a message at the root of its filesystem and keep it extracted
$ ./ubuntu.run --keep --rw tee /message <<< "Hello World"

# Display the message inside the bundle root filesystem
$ enroot start hello-world cat /message
```
23 changes: 23 additions & 0 deletions doc/cmd/create.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Usage

```
Usage: enroot create [options] [--] IMAGE
Create a container root filesystem from a container image.
Options:
-n, --name Name of the container (defaults to "IMAGE")
```

# Description

Take a container image and unpack its root filesystem under `$ENROOT_DATA_PATH/`.
The resulting root filesystem can be started with the [start](start.md) command or removed with the [remove](remove.md) command.

# Example

```sh
# Import Ubuntu 18.04 from DockerHub and create a container out of it
$ enroot import docker://ubuntu:18.04
$ enroot create --name ubuntu ubuntu+18.04.sqsh
```
Loading

0 comments on commit 2c410f2

Please sign in to comment.