feat(erofs): initial commit for erofs support #626

rchincha · 2024-06-14T05:08:51Z

Fixes opencontainers/image-spec#1190

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

hsiangkao · 2024-06-14T05:41:32Z

Hi @rchincha, I'm currently working on incremental builds and I plan to release erofs-utils v1.8 this month (with multi-threaded mkfs, incremental builds, Intel QPL, etc.), but anyway, I'd suggest use cgo for initial EROFS support (with formal liberofs APIs exported) for now (or even binary integration as the very first step) since this go implemention seems somewhat incomplete (and maybe even broken.. I don't have enough time to look into that since other prioritied stuffs are on hands..)

rchincha · 2024-06-14T05:57:35Z

@hsiangkao I suspect that cgo path may be what ends up happening. Currently, just prototyping quickly to understand the interfaces and scope of changes/work.

hsiangkao · 2024-09-18T01:41:08Z

One thing I might need to mention here is that erofs supports external blobs or chunks since Linux 5.16 compared to Squashfs because Dragonfly Nydus once asked this feature to deduplicate data in chunks among different container images.

I'm not sure if it's worthwhile to highlight this, you could just make a tiny metadata with external blobs (which can be used for multiple images) for reference.
Currently only Nydus has userspace tools to generate chunk blobs, if that is interested in other use cases, I could seek more time to implement this in mkfs.erofs too.

rchincha · 2024-11-16T06:10:05Z

$ cat stacker.yaml

build:
  from:
    type: docker
    url: docker://public.ecr.aws/docker/library/busybox:1.37-glibc

 $ /tmp/stacker  build -layer-type erofs
Creating new OCI Layout at "/data/hdd/rchincha/tmp/lxc/oci"
preparing image build...
imported file hashes (after substitutions):
loading docker://public.ecr.aws/docker/library/busybox:1.37-glibc
Copying blob a46fbb00284b done
Copying config 27a71e19c9 done
Writing manifest to image destination
Storing signatures
mkfs.erofs 1.4                                                                                                                                                                    c_version:           [     1.4]
        c_dbg_lvl:           [       2]
        c_dry_run:           [       0]
filesystem build built successfully

rchincha · 2024-11-18T20:53:23Z

@hsiangkao an update. Now have a stacker (this PR) that can build OCI images with erofs+dm-verity layers - no change to the overall workflow.

codecov · 2024-11-18T21:40:52Z

Codecov Report

Attention: Patch coverage is 8.10811% with 34 lines in your changes missing coverage. Please review.

Project coverage is 6.65%. Comparing base (ad9a694) to head (ba91142).

Files with missing lines	Patch %	Lines
pkg/types/layer_type.go	0.00%	20 Missing ⚠️
pkg/overlay/pack.go	0.00%	6 Missing ⚠️
cmd/stacker/internal_go.go	0.00%	4 Missing ⚠️
cmd/stacker/validate.go	0.00%	2 Missing ⚠️
cmd/stacker/build.go	75.00%	1 Missing ⚠️
cmd/stacker/publish.go	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (ad9a694) and HEAD (ba91142). Click for more details.

HEAD has 43 uploads less than BASE

Flag BASE (ad9a694) HEAD (ba91142)

44 1

Additional details and impacted files

@@            Coverage Diff            @@
##             main    #626      +/-   ##
=========================================
- Coverage   13.08%   6.65%   -6.43%     
=========================================
  Files          35      57      +22     
  Lines        5084    6457    +1373     
=========================================
- Hits          665     430     -235     
- Misses       4309    5964    +1655     
+ Partials      110      63      -47

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

atomfs has added support for additional filesystem types such as erofs. Signed-off-by: Ramkumar Chinchani <[email protected]>

rchincha · 2024-11-21T19:38:07Z

@hsiangkao

Is there an equivalent of "unsquashfs" in erofs-utils?

https://manpages.debian.org/testing/squashfs-tools/unsquashfs.1.en.html

SYNOPSIS

unsquashfs [OPTIONS] FILESYSTEM [files to extract or exclude (with -excludes) or cat (with -cat )]

hsiangkao · 2024-11-22T02:34:50Z

Hi!

@hsiangkao

Is there an equivalent of "unsquashfs" in erofs-utils?

https://manpages.debian.org/testing/squashfs-tools/unsquashfs.1.en.html

SYNOPSIS

unsquashfs [OPTIONS] FILESYSTEM [files to extract or exclude (with -excludes) or cat (with -cat )]

fsck.erofs --extract=<dir> FILESYSTEM to extract the whole filesystem...

Currently, there is no exclude way but it can be added later.

Signed-off-by: Ramkumar Chinchani <[email protected]>

raharper · 2024-12-09T03:29:44Z

install-build-deps.sh

@@ -43,6 +43,7 @@ installdeps_ubuntu() {
        squashfs-tools
        squashfuse
        libarchive-tools
+        erofs-utils erofsfuse


one per line

raharper · 2024-12-09T03:31:53Z

pkg/types/layer_type.go

+		fallthrough
+	case squashfs.GenerateSquashfsMediaType(squashfs.ZstdCompression):
+		return NewLayerType("squashfs", verity.VerityMetadataMissing)
+	case squashfs.GenerateSquashfsMediaType(squashfs.GzipCompression):


We already had a case for Squash + Gzip -- was this supposed to be zstd?

raharper · 2024-12-09T03:32:55Z

pkg/types/layer_type.go

+	case erofs.GenerateErofsMediaType(erofs.LZ4HCCompression):
+		return NewLayerType("erofs", verity.VerityMetadataMissing)
+	case erofs.GenerateErofsMediaType(erofs.LZ4HCCompression):


These are the same case right? LZ4H , fallthrough?

raharper · 2024-12-09T03:34:36Z

test/atomfs-erofs.bats

+    manifest=$(cat oci/index.json | jq -r .manifests[0].digest | cut -f2 -d:)
+    last_layer_num=$(($(cat oci/blobs/sha256/$manifest | jq -r '.layers | length')-1))
+    last_layer_hash=$(cat oci/blobs/sha256/$manifest | jq -r .layers[$last_layer].digest | cut -f2 -d:)


just a general comment, prolly worth an issue, but we can avoid the cat | jq and just jq xxx < file

raharper · 2024-12-09T03:36:43Z

test/helpers.bash

+      return
+   fi
+   if [ -z "$SUDO_UID" ]; then
+      echo "PRIVILEGE_LEVEL=$PRIVILEGE_LEVEL but empty SUDO_USER"


SUDO_USER or SUDO_UID ? both are present, but your error prints a different ENV value than what is checked.

Suggested change

echo "PRIVILEGE_LEVEL=$PRIVILEGE_LEVEL but empty SUDO_USER"

echo "PRIVILEGE_LEVEL=$PRIVILEGE_LEVEL but empty SUDO_UID"

raharper · 2024-12-09T03:38:18Z

test/helpers.bash

+   if [ "$PRIVILEGE_LEVEL" = "priv" ]; then
+      return
+   fi
+   if [ -z "$SUDO_UID" ]; then


Suggested change

if [ -z "$SUDO_UID" ]; then

if [ -z "$SUDO_USER" ]; then

which one? SUDO_UID or SUDO_USER?

rchincha requested review from rchamarthy, raharper, mikemccracken and ariel-miculas June 14, 2024 05:09

rchincha force-pushed the erofs branch from 449e5f5 to a6b5f8c Compare June 14, 2024 05:11

rchincha force-pushed the erofs branch from a6b5f8c to 9cbbb3e Compare August 14, 2024 18:33

VPS6123456 approved these changes Aug 16, 2024

View reviewed changes

rchincha force-pushed the erofs branch 3 times, most recently from c700aaa to 762bfcb Compare August 29, 2024 22:39

ariel-miculas mentioned this pull request Sep 23, 2024

I suggest doing speed/size comparison with other dedup. solutions project-machine/puzzlefs#114

Open

rchincha force-pushed the erofs branch from 762bfcb to 284c7f5 Compare November 10, 2024 07:46

rchincha mentioned this pull request Nov 10, 2024

feat: initial commit for erofs support project-machine/atomfs#25

Open

rchincha force-pushed the erofs branch 2 times, most recently from 7814fb5 to 7f7504c Compare November 16, 2024 06:02

rchincha force-pushed the erofs branch from 7f7504c to f92d0fd Compare November 16, 2024 23:13

rchincha force-pushed the erofs branch 2 times, most recently from 31b8397 to ea8d893 Compare November 18, 2024 21:29

rchincha force-pushed the erofs branch from ea8d893 to dc35a77 Compare November 18, 2024 21:50

feat: add support for erofs layers

ca4e173

atomfs has added support for additional filesystem types such as erofs. Signed-off-by: Ramkumar Chinchani <[email protected]>

rchincha force-pushed the erofs branch 4 times, most recently from 460e2de to 77e6c38 Compare November 20, 2024 07:57

rchincha force-pushed the erofs branch from 8266ea9 to b189f0a Compare November 21, 2024 04:03

rchincha force-pushed the erofs branch 3 times, most recently from 7d76f40 to ffe1908 Compare November 21, 2024 23:16

rchincha mentioned this pull request Nov 22, 2024

performance: what can image-spec do to improve handling of large images? opencontainers/image-spec#1190

Open

rchincha force-pushed the erofs branch from ffe1908 to ba91142 Compare November 23, 2024 03:01

test: add a erofs test

8d9b9c6

Signed-off-by: Ramkumar Chinchani <[email protected]>

rchincha force-pushed the erofs branch from ba91142 to 8d9b9c6 Compare November 28, 2024 07:24

rchincha added 2 commits December 2, 2024 05:52

add

bf3be76

fix: media-types are fixed

9b9e659

raharper requested changes Dec 9, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(erofs): initial commit for erofs support #626

feat(erofs): initial commit for erofs support #626

rchincha commented Jun 14, 2024

hsiangkao commented Jun 14, 2024 •

edited

Loading

rchincha commented Jun 14, 2024

hsiangkao commented Sep 18, 2024 •

edited

Loading

rchincha commented Nov 16, 2024

rchincha commented Nov 18, 2024

codecov bot commented Nov 18, 2024 •

edited

Loading

rchincha commented Nov 21, 2024

hsiangkao commented Nov 22, 2024 •

edited

Loading

raharper Dec 9, 2024

raharper Dec 9, 2024

raharper Dec 9, 2024

raharper Dec 9, 2024

raharper Dec 9, 2024

raharper Dec 9, 2024

	echo "PRIVILEGE_LEVEL=$PRIVILEGE_LEVEL but empty SUDO_USER"
	echo "PRIVILEGE_LEVEL=$PRIVILEGE_LEVEL but empty SUDO_UID"

feat(erofs): initial commit for erofs support #626

Are you sure you want to change the base?

feat(erofs): initial commit for erofs support #626

Conversation

rchincha commented Jun 14, 2024

hsiangkao commented Jun 14, 2024 • edited Loading

rchincha commented Jun 14, 2024

hsiangkao commented Sep 18, 2024 • edited Loading

rchincha commented Nov 16, 2024

rchincha commented Nov 18, 2024

codecov bot commented Nov 18, 2024 • edited Loading

Codecov Report

rchincha commented Nov 21, 2024

hsiangkao commented Nov 22, 2024 • edited Loading

raharper Dec 9, 2024

Choose a reason for hiding this comment

raharper Dec 9, 2024

Choose a reason for hiding this comment

raharper Dec 9, 2024

Choose a reason for hiding this comment

raharper Dec 9, 2024

Choose a reason for hiding this comment

raharper Dec 9, 2024

Choose a reason for hiding this comment

raharper Dec 9, 2024

Choose a reason for hiding this comment

hsiangkao commented Jun 14, 2024 •

edited

Loading

hsiangkao commented Sep 18, 2024 •

edited

Loading

codecov bot commented Nov 18, 2024 •

edited

Loading

hsiangkao commented Nov 22, 2024 •

edited

Loading