erofs: Change the erofs compression default to zstd

[bcl]

--rootfs-type erofs now uses:
zstd,8 -Ededupe,all-fragments -C 65536

unless overridden by the 'compression.erofs' section of a lorax config file.

[coveralls]

Pull Request Test Coverage Report for Build 11939805641

Details

• 3 of 5 (60.0%) changed or added relevant lines in 3 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 43.507%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/pylorax/init.py	0	2	0.0%

Totals
Change from base Build 11445659534:	0.0%
Covered Lines:	1636
Relevant Lines:	3560

---

💛 - Coveralls

[Conan-Kudo]

What's the justification for -Ededupe,all-fragments -C 65536?

[bcl]

It produces somewhat smaller images. They're still not really close to squashfs yet, but it is an improvement.

[hsiangkao]

hi @bcl and @Conan-Kudo, I've added a FAQ for this https://erofs.docs.kernel.org/en/latest/faq.html

Anyway, you could also package all files into tar, and compress this tar with EROFS and Squashfs to confirm if the difference is from metadata compression then.

-C65536 or -C131072 is a matchable configuration for Squashfs default -b131072.

If you compares with Squashfs xz images, I guess -zlzma -C65536 -Eall-fragments may be a better choice since -Ededupe doesn't support multi-threaded compression for now so it will be slow. EROFS doesn't support BCJ filters but it's on our roadmap to support BCJ filters for any EROFS supported algorithms in addtition to lzma.

[Conan-Kudo]

I'm trying out zstd,1 with the rest of those options right now: https://pagure.io/fedora-kiwi-descriptions/pull-request/105

[Conan-Kudo]

Okay -Ededupe is way too slow. It's causing all CI runs to time out, and even my local runs are 40+ minutes.

[Conan-Kudo]

With squashfs+xz, my Fedora KDE Rawhide live ISO is approximately 2.7GB. The same image with erofs+zstd:1+all-fragments+chunk:131072 is approximately 3.6GB. With erofs+zstd:12+all-fragments+chunk:131072, it is approximately 3.3GB. With erofs+lzma:6+all-fragments+chunk:131072, it is approximately 3.0GB.

[hsiangkao]

Okay -Ededupe is way too slow. It's causing all CI runs to time out, and even my local runs are 40+ minutes.

that is a problem due to -Ededupe is still single-threaded...

[hsiangkao]

With squashfs+xz, my Fedora KDE Rawhide live ISO is approximately 2.7GB. The same image with erofs+zstd:1+all-fragments+chunk:131072 is approximately 3.6GB. With erofs+zstd:12+all-fragments+chunk:131072, it is approximately 3.3GB. With erofs+lzma:6+all-fragments+chunk:131072, it is approximately 3.0GB.

I guess erofs+lzma:6+all-fragments+dudupe+chunk:131072 will be smaller, since I vaguely remembered I tested it for fedora images before (although it takes many hours), but it's still too slow for now and I need time to implement the multithreading for this.

I'm not sure if ~0.3GB is all due to metadata compression, but you could try to generate a tar for all files first, and then compress the tar with squashfs+xz+bcj and erofs+lzma:6+all-fragments+chunk:131072. If the result closes, it's due to metadata compression, if not, that is due to lack of erofs lzma bcj support...

Or you could use mksquashfs -noI option to disable its metadata compression and drop -Xbcj to confirm this too..

[Conan-Kudo]

I didn't have a way to turn off -Xbcj in kiwi, but I was able to set -noI and it made no appreciable change in squashfs size.

[hsiangkao]

@Conan-Kudo Anyway that is the current status of the image sizes, if you have more time to confirm erofs+lzma:6+all-fragments+dudupe+chunk:131072 and assume that is smaller than the previous squashfs image, I will try to focus on multithreaded compressed deduplication support for this.
If not, it would be very helpful to know the image size gap is due to metadata compression or BCJ. BCJ is just like a pre/postprocessing binary converter to get better compression for executable files. I have some ongoing work for generic BCJ, and at least I tested lz4 BCJ, it produces smaller images too.

In short, I know there is still efforts to close the image size gap against squashfs. and I will try to prioritize some first, maybe multithreaded compressed deduplication support is more useful but it'd be better to confirm (even it's too slow..)

[Conan-Kudo]

@Conan-Kudo Anyway that is the current status of the image sizes, if you have more time to confirm erofs+lzma:6+all-fragments+dudupe+chunk:131072 and assume that is smaller than the previous squashfs image, I will try to focus on multithreaded compressed deduplication support for this.

I will do a run with this, I do expect it to take multiple hours though.

[hsiangkao]

I didn't have a way to turn off -Xbcj in kiwi, but I was able to set -noI and it made no appreciable change in squashfs size.

I guess could you try to disable this by dropping

lorax/src/pylorax/creator.py

Line 128 in 55701a1

compressargs = ["-Xbcj", arch.bcj]

?

[Conan-Kudo]

I'm not using lorax, I'm using kiwi, but yes I could do something similar.

[bcl]

Please move kiwi discussions to a more appropriate place :)

FWIW in my tests I can use -Ededupe,all-fragments without massive slowdown. On my laptop it takes about 5 minutes to compress the install.img using zstd,8

Trying 128k block size saved 4MB, using -T only saved 1MB and using zstd,12 doubled the time to 11 minutes so for now I'm leaving the settings as-is in lorax, they seem to be a reasonable place to start for people who want to use it.

[hsiangkao]

Please move kiwi discussions to a more appropriate place :)

FWIW in my tests I can use -Ededupe,all-fragments without massive slowdown. On my laptop it takes about 5 minutes to compress the install.img using zstd,8

Trying 128k block size saved 4MB, using -T only saved 1MB and using zstd,12 doubled the time to 11 minutes so for now I'm leaving the settings as-is in lorax, they seem to be a reasonable place to start for people who want to use it.

yes, zstd is faster with -Ededupe but zstd is still not comparable in size against lzma...

[Conan-Kudo]

Please move kiwi discussions to a more appropriate place :)

FWIW in my tests I can use -Ededupe,all-fragments without massive slowdown. On my laptop it takes about 5 minutes to compress the install.img using zstd,8

Trying 128k block size saved 4MB, using -T only saved 1MB and using zstd,12 doubled the time to 11 minutes so for now I'm leaving the settings as-is in lorax, they seem to be a reasonable place to start for people who want to use it.

That's odd, because it is very slow on my Framework 16, and it's also quite slow in Zuul CI. It's not specific to kiwi, as it's the actual mkfs.erofs run that is slow.

[Conan-Kudo]

@Conan-Kudo Anyway that is the current status of the image sizes, if you have more time to confirm erofs+lzma:6+all-fragments+dudupe+chunk:131072 and assume that is smaller than the previous squashfs image, I will try to focus on multithreaded compressed deduplication support for this.

I will do a run with this, I do expect it to take multiple hours though.

Anyway, this run took 1 hour and 15 minutes, but the resulting image is 2.8GB.

[hsiangkao]

@Conan-Kudo Anyway that is the current status of the image sizes, if you have more time to confirm erofs+lzma:6+all-fragments+dudupe+chunk:131072 and assume that is smaller than the previous squashfs image, I will try to focus on multithreaded compressed deduplication support for this.

I will do a run with this, I do expect it to take multiple hours though.

Anyway, this run took 1 hour and 15 minutes, but the resulting image is 2.8GB.

@Conan-Kudo ok, then I assume both dedupe and BCJ can help to get a smaller image then..

Also it seems kiwi uses -b 1m to compress?
https://github.com/OSInside/kiwi/blob/8b14c13806a3f27d3334ef1a786bbfe1e995da41/kiwi/filesystem/squashfs.py#L76
so EROFS should also use -C 1048576... like -zlzma,6 1048576 -Eall-fragments,dedupe, anyway...

[Conan-Kudo]

@Conan-Kudo Anyway that is the current status of the image sizes, if you have more time to confirm erofs+lzma:6+all-fragments+dudupe+chunk:131072 and assume that is smaller than the previous squashfs image, I will try to focus on multithreaded compressed deduplication support for this.

I will do a run with this, I do expect it to take multiple hours though.

Anyway, this run took 1 hour and 15 minutes, but the resulting image is 2.8GB.

@Conan-Kudo ok, then I assume both dedupe and BCJ can help to get a smaller image then..

Yeah, I think so. I'd like to get to the point where we can get compression parity and set similar defaults in kiwi like lorax is attempting to do now.