fix(umoci)!: umoci's copy buffer set to 1 MiB (#537)

* ci: add rules for 'rel-**' branches

Signed-off-by: Ramkumar Chinchani <[email protected]>
Signed-off-by: Scott Moser <[email protected]>

* chore: Update squashfs dependency, pin squashfs-tools-ng to 1.2

anuvu/squashfs will not currently build with master from
squashfs-tools-ng.  squashfs-tools-ng has just moved forward with
breaking changes to their api.

We need to build and use a specific version.

Change the squashfs-tools-ng both on the host
(.github/workflows/build.yaml) and in the stacker
build.yaml file

Signed-off-by: Scott Moser <[email protected]>

* ci: drop unnecessary build of anuvu/squashfs binary

This step is just not necessary, we do not use the 'squashfs' binary
for anything.

Signed-off-by: Scott Moser <[email protected]>

* fix: update build.yaml to address bitrot.

This branch has not been built for some time.  In that time, alpine
container has moved forward, and has caused bitrot.  The changes
here are all present in main branch's build.yaml.

Signed-off-by: Scott Moser <[email protected]>

* fix: build.yaml run section needs set -e, and add -x

Mostly a cherry-pick of 3134d84
But we do not need the po4a change here as we have an older
version of xz.

Signed-off-by: Scott Moser <[email protected]>

* fix: update and re-enable test "test can read previous version's cache"

cherry-pick commments:
> below, is original upstream commit message.
> Things that have been adjusted for this branch are:
> * There is no 're-enable' here as it was not disabled here
> * The version to compare against is v0.40.1, the last released
>   version on this branch.

Commit 9671d8f disabled the caching.bats test
"test can read previous version's cache".

This re-enables that test after updating it to use a specific value
(v1.0.0-rc4) as the "previous version".

The change to use a specific version and use its released binary
improves the test. That improves the test for the following
reasons:

 * This should be faster - we don't have to build stacker twice.
 * Ultimately testing against the last released version is probably what
   you want... that is the cache that your users have, not the last
   development commit.
 * Building stacker from another version is both slow and hard to
   guarantee. This removes the assumption that the current build
   environment can build another version of stacker.
 * There was assumed knowledge about both the current and other stacker
   build systems in the 'make LXC_BRANCH=$(grep ...)' command.
 * The prior version of the test used "current/main" as "previous", but
   current/main is a moving target.  Re-running a checkout
   at a future point in time may fail as current/main has moved forward.
   (Think about using 'git bisect' to diagnose a past regression)
 * Even when working as intended, a series of PRs could still break
   re-use of the cache.  Given PRs A, B, C that all are merged to
   master.  B is tested against A, C is tested against B, but
   nothing tested C against A.

The change here *does* mean the following things are true, which are
possibly undesirable:

 * There is a requirement to update the old version string.
   v1.0.0-rc4 is an arbitrary value.
 * The old binary must run in the current build environment.  This
   may be hard to guarantee.  I'm honestly not sure if static binary
   will be more reliable than building and executing a specific old
   version of source.

(cherry picked from commit b4f7294)
Signed-off-by: Scott Moser <[email protected]>

* fix(umoci)!: umoci's copy buffer set to 1 MiB

project-stacker/umoci@89a7ea1

Use updated umoci (project-stacker's fork) to use a 1 MiB copy buffer.
This change brings stacker inline with compression parameters used in
containers/image libraries and makes them compatible.

* test(gzip): add unit tests for gzip

(cherry picked from commit 589a648)

Signed-off-by: Ramkumar Chinchani <[email protected]>
Signed-off-by: Scott Moser <[email protected]>

---------

Signed-off-by: Ramkumar Chinchani <[email protected]>
Signed-off-by: Scott Moser <[email protected]>
Co-authored-by: Ramkumar Chinchani <[email protected]>

.github/workflows/build.yaml

@@ -35,12 +35,26 @@ jobs:
         run: |
           sudo apt-get update
           sudo apt-get install -yy lxc-utils lxc-dev libacl1-dev jq libcap-dev libseccomp-dev libpam-dev bats parallel libzstd-dev
+          # skopeo deps
+          sudo apt-get install -yy \
+             libgpgme-dev \
+             libassuan-dev \
+             libbtrfs-dev \
+             libdevmapper-dev \
+             pkg-config
           GO111MODULE=off go get github.com/opencontainers/umoci/cmd/umoci
           sudo cp ~/go/bin/umoci /usr/bin
           curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin
           sudo apt-get install -yy autoconf automake make autogen autoconf libtool binutils git squashfs-tools libcryptsetup-dev libdevmapper-dev cryptsetup-bin squashfuse
-          (cd /tmp && git clone https://github.com/AgentD/squashfs-tools-ng && cd squashfs-tools-ng && ./autogen.sh && ./configure --prefix=/usr && make -j2 && sudo make -j2 install && sudo ldconfig -v)
-          (cd /tmp && git clone https://github.com/anuvu/squashfs && cd squashfs && make && sudo cp squashtool/squashtool /usr/bin)
+          (cd /tmp &&
+              git clone https://github.com/AgentD/squashfs-tools-ng &&
+              cd squashfs-tools-ng &&
+              git checkout origin/fixes-1.2.0
+              ./autogen.sh &&
+              ./configure --prefix=/usr &&
+              make -j2 &&
+              sudo make -j2 install &&
+              sudo ldconfig -v)
       - if: github.event_name != 'release' || github.event.action != 'published'
         name: Build and test
         run: |

.github/workflows/ci.yaml

@@ -4,9 +4,11 @@ on:
   push:
     branches:
       - main
+      - rel-**
   pull_request:
     branches:
       - main
+      - rel-**
 
 jobs:
   build:

.github/workflows/cloc.yml

@@ -3,9 +3,11 @@ on:
   push:
     branches:
       - main
+      - rel-**
   pull_request:
     branches:
       - main
+      - rel-**
   release:
     types:
       - published

.github/workflows/codeql-analysis.yml

@@ -13,10 +13,10 @@ name: "CodeQL"
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, rel-** ]
   pull_request:
     # The branches below must be a subset of the branches above
-    branches: [ main ]
+    branches: [ main, rel-** ]
   schedule:
     - cron: '17 11 * * 0'
 

.github/workflows/commit-msg.yaml

@@ -9,6 +9,7 @@ on:
   push:
     branches:
       - main
+      - rel-**
 
 jobs:
   check-commit-message-style:

.github/workflows/dco.yml

@@ -4,6 +4,7 @@ on:
   pull_request:
     branches:
       - main
+      - rel-**
 
 permissions: read-all
 

Makefile

@@ -2,6 +2,9 @@ GO_SRC=$(shell find . -path ./.build -prune -false -o -name \*.go)
 VERSION?=$(shell git describe --tags || git rev-parse HEAD)
 VERSION_FULL?=$(if $(shell git status --porcelain --untracked-files=no),$(VERSION)-dirty,$(VERSION))
 
+export SKOPEO = $(TOOLS_D)/bin/skopeo
+export SKOPEO_VERSION = 1.9.3
+
 LXC_VERSION?=$(shell pkg-config --modversion lxc)
 
 BUILD_TAGS = exclude_graphdriver_btrfs exclude_graphdriver_devicemapper containers_image_openpgp osusergo netgo
@@ -44,13 +47,26 @@ lint: cmd/stacker/lxc-wrapper/lxc-wrapper $(GO_SRC)
 	go test -tags "$(BUILD_TAGS)" ./...
 	$(shell go env GOPATH)/bin/golangci-lint run --build-tags "$(BUILD_TAGS)"
 
+$(SKOPEO):
+	@mkdir -p "$(TOOLS_D)/bin"; \
+	tmpdir=$$(mktemp -d); \
+	cd $$tmpdir; \
+	git clone https://github.com/containers/skopeo.git; \
+	cd skopeo; \
+	git fetch --all --tags --prune; \
+	git checkout tags/v$(SKOPEO_VERSION) -b tag-$(SKOPEO_VERSION); \
+	make bin/skopeo; \
+	cp bin/skopeo $(SKOPEO); \
+	cd $(TOP_LEVEL); \
+	rm -rf $$tmpdir;
+
 TEST?=$(patsubst test/%.bats,%,$(wildcard test/*.bats))
 PRIVILEGE_LEVEL?=
 
 # make check TEST=basic will run only the basic test
 # make check PRIVILEGE_LEVEL=unpriv will run only unprivileged tests
 .PHONY: check
-check: stacker lint
+check: stacker lint $(SKOPEO)
 	sudo -E PATH="$$PATH" \
 		LXC_BRANCH=$(LXC_BRANCH) \
 		LXC_CLONE_URL=$(LXC_CLONE_URL) \

build.yaml

@@ -8,22 +8,21 @@ build-env:
     - https://gitlab.com/cryptsetup/cryptsetup/-/archive/v2.4.3/cryptsetup-v2.4.3.tar.gz
     - https://github.com/lvmteam/lvm2/archive/refs/tags/v2_03_15.tar.gz
   run: |
-    #!/bin/sh
+    #!/bin/sh -ex
     # libapparmor is only in testing
     head -n1 /etc/apk/repositories | sed 's/main/testing/g' >> /etc/apk/repositories
 
     apk add git findutils go automake autoconf make gcc libtool \
-      acl-dev acl-static \
+      acl-dev acl-static build-base \
       libseccomp-dev libseccomp-static \
       libcap-dev libcap-static \
       libapparmor-dev \
       zlib-static lz4-static \
-      zstd-dev \
+      zstd-dev zstd-static \
       xz \
       gettext-dev \
-      lvm2-dev util-linux-dev \
-      squashfs-tools-ng-dev \
-      linux-headers
+      lvm2-dev util-linux-dev util-linux-static \
+      linux-headers po4a
 
     # json-c doesn't have static binaries in alpine
     apk add cmake
@@ -44,11 +43,12 @@ build-env:
 
     # build static cryptsetup without all the command line tools
     apk add gettext gettext-dev zlib-static lz4-static openssl-dev \
-      openssl-libs-static popt-dev
+      openssl-libs-static popt-dev bash
     tar -xf /stacker/cryptsetup*
     cd cryptsetup*
     ./autogen.sh
-    ./configure --enable-static \
+    CFLAGS="-D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE" \
+      ./configure --enable-static \
       --disable-cryptsetup --disable-veritysetup --disable-integritysetup \
       --disable-nls --disable-ssh-token
     make -j$(grep -c processor /proc/cpuinfo) install
@@ -72,6 +72,15 @@ build-env:
     make -j$(grep -c processor /proc/cpuinfo) install
     cd /
 
+    # build squashfs-tools-ng at origin/fixes-1.2.0
+    git clone https://github.com/AgentD/squashfs-tools-ng
+    cd squashfs-tools-ng
+    git checkout origin/fixes-1.2.0
+    ./autogen.sh
+    ./configure --prefix=/usr
+    make -j2
+    make -j2 install
+
 build:
   build_only: true
   from:
@@ -80,7 +89,7 @@ build:
   binds:
     - . -> /stacker-tree
   run: |
-    #!/bin/sh
+    #!/bin/sh -ex
     # golang wants somewhere to put its garbage
     export HOME=/root
     export GOPATH=/stacker-tree/.build/gopath

go.mod

@@ -4,7 +4,7 @@ go 1.19
 
 require (
 	github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
-	github.com/anuvu/squashfs v0.0.0-20220228153512-67c8ca12cbf4
+	github.com/anuvu/squashfs v0.0.4
 	github.com/apex/log v1.9.0
 	github.com/cheggaaa/pb/v3 v3.1.0
 	github.com/containers/image/v5 v5.16.1
@@ -23,24 +23,25 @@ require (
 	github.com/pkg/errors v0.9.1
 	github.com/pkg/xattr v0.4.9
 	github.com/smartystreets/goconvey v1.7.2
-	github.com/stretchr/testify v1.8.1
+	github.com/stretchr/testify v1.8.4
 	github.com/twmb/algoimpl v0.0.0-20170717182524-076353e90b94
 	github.com/udhos/equalfile v0.3.0
-	github.com/urfave/cli v1.22.10
-	github.com/vbatts/go-mtree v0.5.2
-	golang.org/x/sys v0.3.0
-	golang.org/x/term v0.3.0
+	github.com/urfave/cli v1.22.12
+	github.com/vbatts/go-mtree v0.5.3
+	golang.org/x/sys v0.14.0
+	golang.org/x/term v0.7.0
 	gopkg.in/yaml.v2 v2.4.0
 )
 
 require (
-	github.com/AdaLogics/go-fuzz-headers v0.0.0-20211102141018-f7be0cbad29c // indirect
-	github.com/BurntSushi/toml v0.4.1 // indirect
+	github.com/AdaLogics/go-fuzz-headers v0.0.0-20230106234847-43070de90fa1 // indirect
+	github.com/BurntSushi/toml v1.2.1 // indirect
 	github.com/Microsoft/go-winio v0.5.2 // indirect
 	github.com/Microsoft/hcsshim v0.9.2 // indirect
 	github.com/VividCortex/ewma v1.2.0 // indirect
 	github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/blang/semver/v4 v4.0.0 // indirect
 	github.com/cespare/xxhash/v2 v2.1.2 // indirect
 	github.com/containerd/cgroups v1.0.3 // indirect
 	github.com/containerd/containerd v1.6.1 // indirect
@@ -56,7 +57,7 @@ require (
 	github.com/docker/docker-credential-helpers v0.6.4 // indirect
 	github.com/docker/go-connections v0.4.0 // indirect
 	github.com/docker/go-metrics v0.0.1 // indirect
-	github.com/docker/go-units v0.4.0 // indirect
+	github.com/docker/go-units v0.5.0 // indirect
 	github.com/fatih/color v1.13.0 // indirect
 	github.com/flosch/pongo2 v0.0.0-20200913210552-0d938eb266f3 // indirect
 	github.com/ghodss/yaml v1.0.0 // indirect
@@ -86,7 +87,7 @@ require (
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/mtrmac/gpgme v0.1.2 // indirect
-	github.com/opencontainers/runc v1.1.2 // indirect
+	github.com/opencontainers/runc v1.1.8 // indirect
 	github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 // indirect
 	github.com/opencontainers/selinux v1.10.0 // indirect
 	github.com/pborman/uuid v1.2.1 // indirect
@@ -96,7 +97,7 @@ require (
 	github.com/prometheus/common v0.32.1 // indirect
 	github.com/prometheus/procfs v0.7.3 // indirect
 	github.com/rivo/uniseg v0.2.0 // indirect
-	github.com/rootless-containers/proto/go-proto v0.0.0-20210921234734-69430b6543fb // indirect
+	github.com/rootless-containers/proto/go-proto v0.0.0-20230421021042-4cd87ebadd67 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/sirupsen/logrus v1.9.0 // indirect
 	github.com/smartystreets/assertions v1.2.0 // indirect
@@ -109,16 +110,18 @@ require (
 	go.etcd.io/bbolt v1.3.6 // indirect
 	go.mozilla.org/pkcs7 v0.0.0-20210826202110-33d05740a352 // indirect
 	go.opencensus.io v0.23.0 // indirect
-	golang.org/x/crypto v0.3.0 // indirect
-	golang.org/x/net v0.2.0 // indirect
+	golang.org/x/crypto v0.8.0 // indirect
+	golang.org/x/net v0.9.0 // indirect
 	golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 // indirect
-	golang.org/x/text v0.4.0 // indirect
+	golang.org/x/text v0.9.0 // indirect
 	google.golang.org/genproto v0.0.0-20220303160752-862486edd9cc // indirect
 	google.golang.org/grpc v1.44.0 // indirect
-	google.golang.org/protobuf v1.28.0 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
 	gopkg.in/robfig/cron.v2 v2.0.0-20150107220207-be2e0b0deed5 // indirect
 	gopkg.in/square/go-jose.v2 v2.5.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
 
 replace github.com/containers/image/v5 => github.com/anuvu/image/v5 v5.0.0-20211117201351-4c24aa76235c
+
+replace github.com/opencontainers/umoci => github.com/project-stacker/umoci v0.0.0-20231025233220-c7bb24d57e98