Buster and 1.9.4 upgrade + bugfixes

.gitignore

@@ -2,8 +2,6 @@ deploy/*
 work/*
 config
 postrun.sh
-SKIP
-SKIP_IMAGES
 .pc
 *-pc
 apt-cacher-ng/

BUILD-GREENGRASS.md

@@ -0,0 +1,14 @@
+# How to build custom Grassbian image?
+
+Place create `./config` file. The sensible minimal one might look like
+the following:
+
+```
+IMG_NAME=CustomGrassbian
+HOSTNAME=grassbian
+```
+
+`HOSTNAME` is optional but it will be randomly-generated otherwise.
+
+If interested in building a full desktop version please remove `SKIP`
+and `SKIP_IMAGES` files from all stages.

Dockerfile

@@ -1,12 +1,12 @@
-FROM debian:stretch
+FROM debian:buster
 
 ENV DEBIAN_FRONTEND noninteractive
 
 RUN apt-get -y update && \
     apt-get -y install \
         git vim parted \
-        quilt realpath qemu-user-static debootstrap zerofree pxz zip dosfstools \
-        bsdtar libcap2-bin rsync grep udev xz-utils curl xxd \
+        quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \
+        bsdtar libcap2-bin rsync grep udev xz-utils curl xxd file kmod\
     && rm -rf /var/lib/apt/lists/*
 
 COPY . /pi-gen/

README.md

@@ -1,6 +1,7 @@
 # Grassbian Users
 
-[use this guide](./SETUP-GREENGRASS.md) to get started
+* [Get started](./SETUP-GREENGRASS.md)
+* [Build own image](./BUILD-GREENGRASS.md)
 
 # pi-gen
 
@@ -10,14 +11,15 @@ _Tool used to create the raspberrypi.org Raspbian images_
 ## Dependencies
 
 pi-gen runs on Debian based operating systems. Currently it is only supported on
-either Debian Stretch or Ubuntu Xenial and is known to have issues building on
-earlier releases of these systems.
+either Debian Buster or Ubuntu Xenial and is known to have issues building on
+earlier releases of these systems. On other Linux distributions it may be possible
+to use the Docker build described below.
 
 To install the required dependencies for pi-gen you should run:
 
 ```bash
-apt-get install quilt parted realpath qemu-user-static debootstrap zerofree pxz zip \
-dosfstools bsdtar libcap2-bin grep rsync xz-utils
+apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \
+dosfstools bsdtar libcap2-bin grep rsync xz-utils file git curl
 ```
 
 The file `depends` contains a list of tools needed.  The format of this
@@ -65,25 +67,86 @@ The following environment variables are supported:
    be built and cached.  Note, `WORK_DIR` stores a complete copy of the target
    system for each build stage, amounting to tens of gigabytes in the case of
    Raspbian.
-   
+
    **CAUTION**: If your working directory is on an NTFS partition you probably won't be able to build. Make sure this is a proper Linux filesystem.
 
  * `DEPLOY_DIR`  (Default: `"$BASE_DIR/deploy"`)
 
    Output directory for target system images and NOOBS bundles.
 
+ * `DEPLOY_ZIP` (Default: `1`)
+
+   Setting to `0` will deploy the actual image (`.img`) instead of a zipped image (`.zip`).
+
  * `USE_QEMU` (Default: `"0"`)
 
    Setting to '1' enables the QEMU mode - creating an image that can be mounted via QEMU for an emulated
    environment. These images include "-qemu" in the image file name.
 
+ * `LOCALE_DEFAULT` (Default: "en_GB.UTF-8" )
+
+   Default system locale.
+
+ * `HOSTNAME` (Default: "raspberrypi" )
+
+   Setting the hostname to the specified value.
+
+ * `KEYBOARD_KEYMAP` (Default: "gb" )
+
+   Default keyboard keymap.
+
+   To get the current value from a running system, run `debconf-show
+   keyboard-configuration` and look at the
+   `keyboard-configuration/xkb-keymap` value.
+
+ * `KEYBOARD_LAYOUT` (Default: "English (UK)" )
+
+   Default keyboard layout.
+
+   To get the current value from a running system, run `debconf-show
+   keyboard-configuration` and look at the
+   `keyboard-configuration/variant` value.
+
+ * `TIMEZONE_DEFAULT` (Default: "Europe/London" )
+
+   Default keyboard layout.
+
+   To get the current value from a running system, look in
+   `/etc/timezone`.
+
+ * `FIRST_USER_NAME` (Default: "pi" )
+
+   Username for the first user
+
+ * `FIRST_USER_PASS` (Default: "raspberry")
+
+   Password for the first user
+
+ * `WPA_ESSID`, `WPA_PASSWORD` and `WPA_COUNTRY` (Default: unset)
+
+   If these are set, they are use to configure `wpa_supplicant.conf`, so that the raspberry pi can automatically connect to a wifi network on first boot.
+
+ * `ENABLE_SSH` (Default: `0`)
+
+   Setting to `1` will enable ssh server for remote log in. Note that if you are using a common password such as the defaults there is a high risk of attackers taking over you RaspberryPi.
+
+ * `STAGE_LIST` (Default: `stage*`)
+
+    If set, then instead of working through the numeric stages in order, this list will be followed. For example setting to `"stage0 stage1 mystage stage2"` will run the contents of `mystage` before stage2. Note that quotes are needed around the list. An absolute or relative path can be given for stages outside the pi-gen directory.
 
 A simple example for building Raspbian:
 
 ```bash
 IMG_NAME='Raspbian'
 ```
 
+The config file can also be specified on the command line as an argument the `build.sh` or `build-docker.sh` scripts.
+
+```
+./build.sh -c myconfig
+```
+
+This is parsed after `config` so can be used to override values set there.
 
 ## How the build process works
 
@@ -132,6 +195,13 @@ It is recommended to examine build.sh for finer details.
 
 ## Docker Build
 
+Docker can be used to perform the build inside a container. This partially isolates
+the build from the host system, and allows using the script on non-debian based
+systems (e.g. Fedora Linux). The isolate is not complete due to the need to use
+some kernel level services for arm emulation (binfmt) and loop devices (losetup).
+
+To build:
+
 ```bash
 vi config         # Edit your config file. See above.
 ./build-docker.sh
@@ -147,6 +217,12 @@ continue:
 CONTINUE=1 ./build-docker.sh
 ```
 
+To examine the container after a failure you can enter a shell within it using:
+
+```bash
+sudo docker run -it --privileged --volumes-from=pigen_work pi-gen /bin/bash
+```
+
 After successful build, the build container is by default removed. This may be undesired when making incremental changes to a customized build. To prevent the build script from remove the container add
 
 ```bash
@@ -253,3 +329,29 @@ follows:
  * Once you're happy with the image you can remove the SKIP_IMAGES files and
    export your image to test
 
+# Troubleshooting
+
+## `64 Bit Systems`
+Please note there is currently an issue when compiling with a 64 Bit OS. See https://github.com/RPi-Distro/pi-gen/issues/271
+
+## `binfmt_misc`
+
+Linux is able execute binaries from other architectures, meaning that it should be
+possible to make use of `pi-gen` on an x86_64 system, even though it will be running
+ARM binaries. This requires support from the [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc)
+kernel module.
+
+You may see the following error:
+
+```
+update-binfmts: warning: Couldn't load the binfmt_misc module.
+```
+
+To resolve this, ensure that the following files are available (install them if necessary):
+
+```
+/lib/modules/$(uname -r)/kernel/fs/binfmt_misc.ko
+/usr/bin/qemu-arm-static
+```
+
+You may also need to load the module by hand - run `modprobe binfmt_misc`.

SETUP-GREENGRASS.md

@@ -6,9 +6,10 @@ You can get the latest image [here](https://github.com/iopipe/pi-gen-greengrass/
 2. Burn the image
 3. Eject and re-insert the card, go into the mounted drive partition (boot partition for Linux afficionados)
 4. Copy the certs/ and config/ folders from the .tar.gz you recieved when you created a greengrass core into the boot partition in the greengrass folder
+5. If default AWS certificate (*Amazon Root CA 1*) is not suitable please copy the one used into the boot partition in `greengrass/certs` folder
 6. (optional) Add a file titled `ssh` with no extension if you want to ssh into your raspberry pi
 7. (optional) [Set up WiFi headlessly](https://www.raspberrypi.org/documentation/configuration/wireless/headless.md)
 7. Eject sd and place it into raspberry pi and apply power to pi
 8. If you haven't set wifi up on the SD card, [do so now](https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md) with an attached keybeard and mouse. Otherwise, you can ssh in if you completed the ssh and headless wifi step.
 8. On the Pi, after boot, run `sudo systemctl status greengrass.service` to ensure greengrass service has booted and is running
-9. If not, `sudo systemctl stop greengrass.service` and `cd /greengrass/ggc/core/`, then `./greengrassd` to see output from startup sequences for troubleshooting
+9. If not, `sudo systemctl stop greengrass.service` and `cd /greengrass/ggc/core/`, then `./greengrassd` to see output from startup sequences for troubleshooting

build-docker.sh

@@ -1,86 +1,108 @@
-#!/bin/bash -e
+#!/bin/bash -eu
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+BUILD_OPTS="$*"
 
 DOCKER="docker"
-set +e
-$DOCKER ps >/dev/null 2>&1
-if [ $? != 0 ]; then
+
+if ! ${DOCKER} ps >/dev/null 2>&1; then
 	DOCKER="sudo docker"
 fi
-if ! $DOCKER ps >/dev/null; then
+if ! ${DOCKER} ps >/dev/null; then
 	echo "error connecting to docker:"
-	$DOCKER ps
+	${DOCKER} ps
 	exit 1
 fi
-set -e
 
-config_file=()
-if [ -f config ]; then
-	config_file=("--env-file" "$(pwd)/config")
-	source config
+CONFIG_FILE=""
+if [ -f "${DIR}/config" ]; then
+	CONFIG_FILE="${DIR}/config"
+fi
+
+while getopts "c:" flag
+do
+	case "${flag}" in
+		c)
+			CONFIG_FILE="${OPTARG}"
+			;;
+		*)
+			;;
+	esac
+done
+
+# Ensure that the configuration file is an absolute path
+if test -x /usr/bin/realpath; then
+	CONFIG_FILE=$(realpath -s "$CONFIG_FILE")
+fi
+
+# Ensure that the confguration file is present
+if test -z "${CONFIG_FILE}"; then
+	echo "Configuration file need to be present in '${DIR}/config' or path passed as parameter"
+	exit 1
+else
+	# shellcheck disable=SC1090
+	source "${CONFIG_FILE}"
 fi
 
 CONTAINER_NAME=${CONTAINER_NAME:-pigen_work}
 CONTINUE=${CONTINUE:-0}
 PRESERVE_CONTAINER=${PRESERVE_CONTAINER:-0}
 
-if [ "$*" != "" ] || [ -z "${IMG_NAME}" ]; then
-	if [ -z "${IMG_NAME}" ]; then
-		echo "IMG_NAME not set in 'config'" 1>&2
-		echo 1>&2
-	fi
-	cat >&2 <<EOF
-Usage:
-    build-docker.sh [options]
-Optional environment arguments: ( =<default> )
-    CONTAINER_NAME=pigen_work  set a name for the build container
-    CONTINUE=1                 continue from a previously started container
-    PRESERVE_CONTAINER=1       keep build container even on successful build
-EOF
-	exit 1
+if [ -z "${IMG_NAME}" ]; then
+	echo "IMG_NAME not set in 'config'" 1>&2
+	echo 1>&2
+exit 1
 fi
 
-CONTAINER_EXISTS=$($DOCKER ps -a --filter name="$CONTAINER_NAME" -q)
-CONTAINER_RUNNING=$($DOCKER ps --filter name="$CONTAINER_NAME" -q)
-if [ "$CONTAINER_RUNNING" != "" ]; then
-	echo "The build is already running in container $CONTAINER_NAME. Aborting."
+# Ensure the Git Hash is recorded before entering the docker container
+GIT_HASH=${GIT_HASH:-"$(git rev-parse HEAD)"}
+
+CONTAINER_EXISTS=$(${DOCKER} ps -a --filter name="${CONTAINER_NAME}" -q)
+CONTAINER_RUNNING=$(${DOCKER} ps --filter name="${CONTAINER_NAME}" -q)
+if [ "${CONTAINER_RUNNING}" != "" ]; then
+	echo "The build is already running in container ${CONTAINER_NAME}. Aborting."
 	exit 1
 fi
-if [ "$CONTAINER_EXISTS" != "" ] && [ "$CONTINUE" != "1" ]; then
-	echo "Container $CONTAINER_NAME already exists and you did not specify CONTINUE=1. Aborting."
+if [ "${CONTAINER_EXISTS}" != "" ] && [ "${CONTINUE}" != "1" ]; then
+	echo "Container ${CONTAINER_NAME} already exists and you did not specify CONTINUE=1. Aborting."
 	echo "You can delete the existing container like this:"
-	echo "  $DOCKER rm -v $CONTAINER_NAME"
+	echo "  ${DOCKER} rm -v ${CONTAINER_NAME}"
 	exit 1
 fi
 
-$DOCKER build -t pi-gen .
-if [ "$CONTAINER_EXISTS" != "" ]; then
-	trap "echo 'got CTRL+C... please wait 5s'; $DOCKER stop -t 5 ${CONTAINER_NAME}_cont" SIGINT SIGTERM
-	time $DOCKER run --rm --privileged \
+# Modify original build-options to allow config file to be mounted in the docker container
+BUILD_OPTS="$(echo "${BUILD_OPTS:-}" | sed -E 's@\-c\s?([^ ]+)@-c /config@')"
+
+${DOCKER} build -t pi-gen "${DIR}"
+if [ "${CONTAINER_EXISTS}" != "" ]; then
+	trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}_cont' SIGINT SIGTERM
+	time ${DOCKER} run --rm --privileged \
+		--volume "${CONFIG_FILE}":/config:ro \
+		-e "GIT_HASH=${GIT_HASH}" \
 		--volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \
-		-e IMG_NAME="${IMG_NAME}"\
 		pi-gen \
 		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
-	cd /pi-gen; ./build.sh;
+	cd /pi-gen; ./build.sh ${BUILD_OPTS} &&
 	rsync -av work/*/build.log deploy/" &
 	wait "$!"
 else
-	trap "echo 'got CTRL+C... please wait 5s'; $DOCKER stop -t 5 ${CONTAINER_NAME}" SIGINT SIGTERM
-	time $DOCKER run --name "${CONTAINER_NAME}" --privileged \
-		-e IMG_NAME="${IMG_NAME}"\
-		"${config_file[@]}" \
+	trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}' SIGINT SIGTERM
+	time ${DOCKER} run --name "${CONTAINER_NAME}" --privileged \
+		--volume "${CONFIG_FILE}":/config:ro \
+		-e "GIT_HASH=${GIT_HASH}" \
 		pi-gen \
 		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
-	cd /pi-gen; ./build.sh &&
+	cd /pi-gen; ./build.sh ${BUILD_OPTS} &&
 	rsync -av work/*/build.log deploy/" &
 	wait "$!"
 fi
 echo "copying results from deploy/"
-$DOCKER cp "${CONTAINER_NAME}":/pi-gen/deploy .
+${DOCKER} cp "${CONTAINER_NAME}":/pi-gen/deploy .
 ls -lah deploy
 
 # cleanup
-if [ "$PRESERVE_CONTAINER" != "1" ]; then
-	$DOCKER rm -v $CONTAINER_NAME
+if [ "${PRESERVE_CONTAINER}" != "1" ]; then
+	${DOCKER} rm -v "${CONTAINER_NAME}"
 fi
 
 echo "Done! Your image(s) should be in deploy/"